How to scrape Bing People Also Ask QnA?

I have created this script by using Simple HTML DOM. The idea behind this script is simple. Open bing search and fetch “People Also Ask” from it.

Here are the steps

Step 1: Include Simple HTML DOM


include('simple_html_dom.php');

it’s an HTML parser for PHP.

Step 2: set the URL to scrape


$url = "https://www.bing.com/search?q=".urlencode('scarlett johansson related questions');

The second step is to set the Bing URL with a search query whose questions u want to fetch.

Step 3: set the User-Agent


ini_set("user_agent","Mozilla/5.0 (Windows NT x.y; Win64; x64; rv:10.0) Gecko/20100101 Firefox/10.0");

The User-Agent request header is a string that identifies the application and operating system to servers and network peers.

Step 4: fetch URL


$html = file_get_html($url);

The actual data comes using the file_get_html function.

Step 5: Parse HTML DOM


foreach($html->find('div[data-tag="RelatedQnA.Item"]') as $element)  {    
    $q = $element->find("div",0)->children(0)->children(0)->children(0)->plaintext;
    $a = $element->find("div",0)->children(1)->children(0)->children(0)->children(0)->plaintext;
    $l = $element->find("div",0)->children(1)->children(0)->children(0)->children(1)->children(0)->children(0)->children(0)->children(0)->getAttribute('href');
    $rec = array("q"=>$q, "a"=>$a, "l"=>$l);
    array_push($array, $rec);
}

By using the browser inspect feature I found where is the exact People QnA location and then fetch all data by putting a loop.

Step 6: CSV File


$out = fopen('file.csv', 'w');
foreach ($array as $fields) {
    fputcsv($out, $fields);
}
fclose($out);

All fetch data is saved in an array, then create a file using fopen function and adding each line in the file using fputcsv function.

Finally,

Just give me source code.

7. Source Code




/// Project: Scrape Bing People also ask from serp result
/// Date: May 4, 2022
/// Time: Morning 7:00 AM
/// Code by Sunil

include('simple_html_dom.php');
$url = "https://www.bing.com/search?q=".urlencode('scarlett johansson related questions');
ini_set("user_agent","Mozilla/5.0 (Windows NT x.y; Win64; x64; rv:10.0) Gecko/20100101 Firefox/10.0");
$html = file_get_html($url);
$array = array();
if($html!=null) {
    foreach($html->find('div[data-tag="RelatedQnA.Item"]') as $element)  {    
        $q = $element->find("div",0)->children(0)->children(0)->children(0)->plaintext;
        $a = $element->find("div",0)->children(1)->children(0)->children(0)->children(0)->plaintext;
        $l = $element->find("div",0)->children(1)->children(0)->children(0)->children(1)->children(0)->children(0)->children(0)->children(0)->getAttribute('href');
        $rec = array("q"=>$q, "a"=>$a, "l"=>$l);
        array_push($array, $rec);
    }
    $out = fopen('file.csv', 'w');
    foreach ($array as $fields) {
        fputcsv($out, $fields);
    }
    fclose($out);   echo "<h1>Well done Sunil</h1><div style='width:480px'<iframe allow='fullscreen' frameBorder='0' height='270' src='https://giphy.com/embed/86hf9kIVHe0bNnUMiG/video' width='480'></iframe></div>";
} else echo "<h1>Opps</h1>";