用幻影和rvest抓取网页
Scrape webpage with phantomjs and rvest
我正在尝试抓取以下网页： https://www.occ.com.mx/empleos-en-nuevo-leon使用rvest和选择器小工具，这似乎很简单。
但是，它似乎是用javascript构建的，所以我按照本教程，安装了phantomjs并尝试使用以下脚本在本地构建html网页。
// scrape_occ.js
var webPage = require('webpage');
var page = webPage.create();
var fs = require('fs');
var path = 'occ.html'
page.open('https://www.occ.com.mx/empleos-en-nuevo-leon', function (status) {
var content = page.content;
fs.write(path,content,'w')
phantom.exit();
});
调用 R： system("phantomjs occ.js")在本地构建 html，但会产生与原始rvest尝试类似的结果。我得到了更多的节点，但似乎有一个形式阻碍了......
p <- read_html("occ.html")
p %>% html_structure()
<html>
<head>
{text}
<meta [http-equiv, content]>
{text}
<meta [http-equiv, content]>
{text}
<meta [http-equiv, content]>
{text}
<noscript>
{text}
{text}
<body>
<apm_do_not_touch>
{text}
<script [language]>
{cdata}
{text}
{text}
<form [action, method]>
<input [type, name, value]>
<input [type, name, value]>
<input [type, name, value]>
<input [type, name, value]>
<input [type, name, value]>
<input [type, name, value]>
<input [type, name, value]>
<input [type, name, value]>
的正确方法是什么？我错过了什么？
RSelenium(对我来说是最后的选择，因为它正在编写浏览器脚本(可以在这里提供帮助：
library(RSelenium)
library(rvest)
library(dplyr)
checkForServer()
startServer()
remDr <- remoteDriver()
remDr$open()
remDr$navigate("https://www.occ.com.mx/empleos-en-nuevo-leon")
pg <- remDr$getPageSource()
remDr$close()
doc <- read_html(pg[[1]])
trimws(html_text(html_nodes(doc, "div.lineamod_sr")))
## [1] "Feb 16 'n Coordinador de Operaciones (Zona Cienega de Flores)"
## [2] "Feb 16 'n Coordinador de Operaciones (Zona Apodaca)"
## [3] "Feb 16 'n Ventas de Tecnologias de informacion (zona Monterrey)"
## [4] "Feb 16 'n Practicante en Capacitación"
## [5] "Feb 16 'n Home Office: Reclutador Bilingüe"
## [6] "Feb 16 'n SISTEMAS"
## [7] "Feb 16 'n Asociado de RH"
## [8] "Feb 16 'n Practicante Import Export"
## [9] "Feb 16 'n Comprador de refacciones"
## [10] "Feb 16 'n Maestra Auxiliar de Preescolar"
在阅读了爱德华多的评论后，我一时兴起，在检查网页并使用"复制为 cURL"(请参阅 Chrome 开发人员工具(获取主页资源后，使用 rvest 尝试了这个。我使用了我的开发工具 pkg curlconverter ( devtools::install_github("hrbrmstr/curlconverter") (
library(httr)
library(rvest)
library(curlconverter)
req <- make_req(straighten("curl 'https://www.occ.com.mx/empleos-en-nuevo-leon' -H 'Pragma: no-cache' -H 'Origin: https://www.occ.com.mx' -H 'Accept-Encoding: gzip, deflate' -H 'Accept-Language: en-US,en;q=0.8' -H 'Upgrade-Insecure-Requests: 1' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.39 Safari/537.36' -H 'Content-Type: application/x-www-form-urlencoded' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8' -H 'Cache-Control: no-cache' -H 'Referer: https://www.occ.com.mx/empleos-en-nuevo-leon' -H 'Cookie: cookie_home3=240784394.20480.0000; TS01650164_28=011b64c78f24f98a905a1610d142f8f5e8e6de557a270318cd246d8bdad634ad9d692ef9a7e7c5f5ee6125dfa6d154a7ecc9912326; q=; cat=; fn=; salary=; tm=60; loc=; page=1; TS01650164_77=08cb11f491ab28002f9f692817ac18f609a6564679dd056ae514dd5ed8f04683e016c84db8ad4aff81249c3ffa6f4d72086a11c1a482380047cb7d1b1a33ef2b110ce9bf57566934a0ba894dd447b3cf48fdfb5b4e17eb905895ebb597a5e878746dd0fba0f4fbb336e2f855820d002a; JobSeekerInfo=5877fd484ea1405b8e4780c2c0b05365|; searchtype=1; loctemp=MX-NL; fntemp=MX-NL; cattemp=; qtemp=; salarytemp=; tmtemp=; jobids=8687636,8687651,8689022,8688254,8688719,8688496,8687786,8687676,8688424,8654486,8688354,8687735,8687291,8688736,8687917,8688614,8687428,8688586,8688659,8687483,8688163,8654471,8687688,8688932,8689017,8642398,8687594,8688554,8688940,8687254,8688285,8415967,8687500,8688224,8687619,8688640,8688757,8689026,8688796,8702349,8728708,8728866,8728860,8721116,8591064,8729151,8729124,8729136,8729485,8551881; returnurl=/empleos-en-nuevo-leon; Culture=es-MX; TS01650164=01b518c62e8b19f2612cfbd7378cc0b73784ab414f17fdcc566da5c243d4a31e6ef629c80498c411d543fa42cb92729a648fc738579c3333ce81ff4708684a7e9f18f32db913b5b26edaa3bb4a0ad4d5ee740b16ab26bce774e39df018e046ddcf1c6529ce4b2194e780f9f039d729db6d958e850ead9570fc4748aa65cd7f85c0b79744cbf998c055a8314887942f194106c71e4cc2f736e495d21f228e9ceabfd36dc194b5ee20d31a00bc2b6469c8645ed502656b60a0a9b38aaf38a3435488f6e942917750d71f27386a87f3ff25f3bd1d163c0e6bd60212d36a3056fbdd4e055dab8c; sitemgrOCCM=false; sitemgrRUE=false; loaded=1' -H 'Connection: keep-alive' -H 'DNT: 1' --data 'TS01650164_id=3&TS01650164_cr=08cb11f491ab2800f5f07209cf7558bf5a68e875ec0c3a7ca8beb59ff0f5da214e8ca0f0b0d3925c3466838922d655d208203848fa894800ea2891c392679d0ca17d55a2a828b5e4bbe97f5e3310800fb216c3a9821974a3aa20fa64d8feec278da6e1000bf2a8ac7f2533a8b0e5f5424e20ea558b261d881d570b85067b4859&TS01650164_76=0&TS01650164_86=0&TS01650164_md=1&TS01650164_rf=0&TS01650164_ct=0&TS01650164_pd=0' --compressed"))[[1]]
resp <- req()
doc <- content(resp, as="parsed")
trimws(html_text(html_nodes(doc, "div.lineamod_sr")))
并且，它返回相同的结果。
curlconverter的东西实际上做了一个这样的函数：
httr::VERB(
verb = "POST",
url = "https://www.occ.com.mx/empleos-en-nuevo-leon",
add_headers(Pragma = "no-cache",
Origin = "https://www.occ.com.mx",
`Accept-Encoding` = "gzip, deflate",
`Accept-Language` = "en-US,en;q=0.8",
`Upgrade-Insecure-Requests` = "1",
`User-Agent` = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.39 Safari/537.36",
Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
`Cache-Control` = "no-cache",
Referer = "https://www.occ.com.mx/empleos-en-nuevo-leon",
Connection = "keep-alive",
DNT = "1"),
set_cookies(cookie_home3 = "240784394.20480.0000",
TS01650164_28 = "011b64c78f24f98a905a1610d142f8f5e8e6de557a270318cd246d8bdad634ad9d692ef9a7e7c5f5ee6125dfa6d154a7ecc9912326",
q = "", cat = "", fn = "",
salary = "", tm = "60",
loc = "", page = "1", TS01650164_77 = "08cb11f491ab28002f9f692817ac18f609a6564679dd056ae514dd5ed8f04683e016c84db8ad4aff81249c3ffa6f4d72086a11c1a482380047cb7d1b1a33ef2b110ce9bf57566934a0ba894dd447b3cf48fdfb5b4e17eb905895ebb597a5e878746dd0fba0f4fbb336e2f855820d002a",
JobSeekerInfo = "5877fd484ea1405b8e4780c2c0b05365|",
searchtype = "1", loctemp = "MX-NL",
fntemp = "MX-NL", cattemp = "",
qtemp = "", salarytemp = "",
tmtemp = "", jobids = "8687636,8687651,8689022,8688254,8688719,8688496,8687786,8687676,8688424,8654486,8688354,8687735,8687291,8688736,8687917,8688614,8687428,8688586,8688659,8687483,8688163,8654471,8687688,8688932,8689017,8642398,8687594,8688554,8688940,8687254,8688285,8415967,8687500,8688224,8687619,8688640,8688757,8689026,8688796,8702349,8728708,8728866,8728860,8721116,8591064,8729151,8729124,8729136,8729485,8551881",
returnurl = "/empleos-en-nuevo-leon",
Culture = "es-MX", TS01650164 = "01b518c62e8b19f2612cfbd7378cc0b73784ab414f17fdcc566da5c243d4a31e6ef629c80498c411d543fa42cb92729a648fc738579c3333ce81ff4708684a7e9f18f32db913b5b26edaa3bb4a0ad4d5ee740b16ab26bce774e39df018e046ddcf1c6529ce4b2194e780f9f039d729db6d958e850ead9570fc4748aa65cd7f85c0b79744cbf998c055a8314887942f194106c71e4cc2f736e495d21f228e9ceabfd36dc194b5ee20d31a00bc2b6469c8645ed502656b60a0a9b38aaf38a3435488f6e942917750d71f27386a87f3ff25f3bd1d163c0e6bd60212d36a3056fbdd4e055dab8c",
sitemgrOCCM = "false",
sitemgrRUE = "false", loaded = "1"),
body = list(TS01650164_id = "3",
TS01650164_cr = "08cb11f491ab2800f5f07209cf7558bf5a68e875ec0c3a7ca8beb59ff0f5da214e8ca0f0b0d3925c3466838922d655d208203848fa894800ea2891c392679d0ca17d55a2a828b5e4bbe97f5e3310800fb216c3a9821974a3aa20fa64d8feec278da6e1000bf2a8ac7f2533a8b0e5f5424e20ea558b261d881d570b85067b4859",
TS01650164_76 = "0", TS01650164_86 = "0",
TS01650164_md = "1", TS01650164_rf = "0",
TS01650164_ct = "0", TS01650164_pd = "0"),
encode = "form")
问题在于，它使用那些与时间相关的 cookie 设置发出POST请求。因此，如果在从开发人员工具会话中提取复制为 cURL 命令后相对较快地完成，这将起作用，但我不确定过了多久。
相关文章：
- 使用htmlunit抓取动态网页
- 使用 HTML5<输入>字段抓取动态生成的网页
- 自动点击按钮和网页抓取
- Python/R中的网页抓取javascript
- 使用Python使用Javascript进行网页抓取
- Python 网页抓取 JavaScript 生成的内容
- 幻影中的变量事务，抓取网页
- 如何在单击按钮时在核心PHP中抓取网页特定区域的屏幕截图
- Python Selenium，抓取网页JavaScript表
- 等待Javascript网页抓取功能完成，然后再运行下一页
- 用幻影和rvest抓取网页
- 保存使用 casperjs 抓取网页时获得的表数据
- 在 c# asp.net 中执行 JavaScript 后抓取网页
- 如何使用jQuery和XPath抓取网页
- 抓取网页并检索javascript变量
- 如何点击“更多”按钮，当Tripadvisor使用selenium抓取网页时
- 无法在Google Apps Script日志中看到完整的抓取网页
- 用python抓取网页以获取onclick值
- 抓取网页没有进行ajax调用，但数据不在DOM中
- 如何在使用BeautifulSoup抓取网页时提取javascript中的内容
