模拟页面点击在HtmlUnit(2.33)给出无效或非法的选择器异常
Simulating page click in HtmlUnit (2.33) gives invalid or illegal selector exception
首先我要说的是，我对Javascript一点也不了解。我试图模拟点击彭博社的超链接页面。我想获取一个新闻项目列表(超链接)，然后简单地遍历该列表，获取每篇文章的标题和文章文本。这是我的代码:
public List<String> getBloomNewsHtmlUnit() throws IOException {
String searchString = "Apple";
List<String> bloombergNewsAll = new ArrayList<>();
WebClient webclient = new WebClient(BrowserVersion.BEST_SUPPORTED);
HtmlPage mainpage = webclient.getPage("http://www.bloomberg.com/search?query=" + searchString);
HtmlAnchor pageanchor = mainpage.getFirstByXPath("//*[@id='"content'"]/div/section/section[2]/section[1]/div[2]/div[2]/article/div[1]/h1/a");
webclient.waitForBackgroundJavaScript(50000);
webclient.getOptions().setThrowExceptionOnScriptError(false);
webclient.getOptions().setThrowExceptionOnFailingStatusCode(false);
webclient.setCssErrorHandler(new SilentCssErrorHandler());
mainpage = pageanchor.click();
System.out.println("Main page: " + mainpage.asText());
return bloombergNewsAll;
// return bloombergNewsAll;
}
这是例外:
Sep 11, 2016 9:49:34 AM com.gargoylesoftware.htmlunit.javascript.StrictErrorReporter runtimeError
SEVERE: runtimeError: message=[An invalid or illegal selector was specified (selector: '*,:x' error: Invalid selector: :x).] sourceName=[https://assets.bwbx.io/business/public/javascripts/application-6e1529c288.js] line=[153] lineSource=[null] lineOffset=[0]
Exception in thread "main" java.lang.RuntimeException: com.gargoylesoftware.htmlunit.ScriptException: TypeError: Cannot call method "split" of undefined (https://assets.bwbx.io/business/public/javascripts/application-6e1529c288.js#79)
at com.gargoylesoftware.htmlunit.html.HtmlPage.initialize(HtmlPage.java:284)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:519)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:386)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:304)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:451)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:436)
at com.jsoup.test.BloombergTest.getBloomNewsHtmlUnit(BloombergTest.java:71)
at com.jsoup.test.BloombergTest.main(BloombergTest.java:37)
Caused by: com.gargoylesoftware.htmlunit.ScriptException: TypeError: Cannot call method "split" of undefined (https://assets.bwbx.io/business/public/javascripts/application-6e1529c288.js#79)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$HtmlUnitContextAction.run(JavaScriptEngine.java:921)
at net.sourceforge.htmlunit.corejs.javascript.Context.call(Context.java:628)
at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.call(ContextFactory.java:515)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.execute(JavaScriptEngine.java:803)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.execute(JavaScriptEngine.java:779)
at com.gargoylesoftware.htmlunit.html.HtmlPage.loadExternalJavaScriptFile(HtmlPage.java:975)
at com.gargoylesoftware.htmlunit.html.HtmlScript.executeScriptIfNeeded(HtmlScript.java:352)
at com.gargoylesoftware.htmlunit.html.HtmlScript$2.execute(HtmlScript.java:238)
at com.gargoylesoftware.htmlunit.html.HtmlPage.initialize(HtmlPage.java:277)
... 7 more
Caused by: net.sourceforge.htmlunit.corejs.javascript.EcmaError: TypeError: Cannot call method "split" of undefined (https://assets.bwbx.io/business/public/javascripts/application-6e1529c288.js#79)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.constructError(ScriptRuntime.java:3915)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.constructError(ScriptRuntime.java:3899)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.typeError(ScriptRuntime.java:3924)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.typeError2(ScriptRuntime.java:3940)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.undefCallError(ScriptRuntime.java:3956)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.getPropFunctionAndThisHelper(ScriptRuntime.java:2390)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.getPropFunctionAndThis(ScriptRuntime.java:2384)
at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpretLoop(Interpreter.java:1342)
at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpret(Interpreter.java:800)
at net.sourceforge.htmlunit.corejs.javascript.InterpretedFunction.call(InterpretedFunction.java:105)
at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.doTopCall(ContextFactory.java:413)
at com.gargoylesoftware.htmlunit.javascript.HtmlUnitContextFactory.doTopCall(HtmlUnitContextFactory.java:252)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3264)
at net.sourceforge.htmlunit.corejs.javascript.InterpretedFunction.exec(InterpretedFunction.java:115)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$3.doRun(JavaScriptEngine.java:794)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$HtmlUnitContextAction.run(JavaScriptEngine.java:906)
... 15 more
Java Result: 1
即使我尝试执行前4行代码(不引用任何html lanchor)，也会出现相同的错误。我在网上读了一些关于这个错误的错误报告，但没有一个建议的解决方案似乎在我的情况下工作:
htmlunit:指定了一个无效或非法的选择器
在上面的SOF问题中，我将建议的waitForBackgroundJavaScript应用到web客户端，但这并没有解决问题。
当点击google结果页时，HtmlUnit中的JavaScript异常
在这个问题中，我试着补充:
JavaScriptEngine engine = webclient.getJavaScriptEngine();
engine.holdPosponedActions();
的代码，但错误仍然在那里。
https://sourceforge.net/p/htmlunit/bugs/1744/在上面的bug报告中，建议使用select查询结果重新定义主页。在我的例子中，我尝试用click()事件重新定义页面。我的代码没有得到那么远，并且在我尝试定义HtmlPage时抛出错误。
https://sourceforge.net/p/htmlunit/bugs/1661/该报告建议简单地忽略警告，但在我的例子中，我得到了一个异常(而不仅仅是警告)，这阻止了期望的输出。
我第一次尝试使用Jsoup来做这个抓取。这工作得很好，但Jsoup在文章文本之间给出了一些错误的链接，这些链接不在原始页面上，当我在Chrome中检查它时。我怀疑是JS或Ajax调用改变了页面DOM。这就是为什么我选择使用Htmlunit。
将感谢任何提示我做错了什么得到这个错误，以及如何纠正它。此外，如果有人认为可以使用Jsoup来实现我想要的东西，请让我知道(我刚刚读到Jsoup不支持DOM中的动态更改，因此无法自行工作)。提前感谢!
该异常并不一定意味着生成的页面是无用的，尽管在其他情况下可能有所不同。你必须检查你要找的内容的结果。
为了减少javascript引擎的错误信息输出，你可以定义:
java.util.logging.Logger.getLogger("com.gargoylesoftware").setLevel(java.util.logging.Level.OFF);
下面的示例选择第一个标题，触发click事件并获取结果页面;为了验证我们是否遵循了链接，打印出标题:
java.util.logging.Logger.getLogger("com.gargoylesoftware").setLevel(java.util.logging.Level.OFF);
final WebClient webClient = new WebClient(BrowserVersion.CHROME);
webClient.getOptions().setCssEnabled(false);
webClient.getOptions().setJavaScriptEnabled(true);
webClient.getOptions().setThrowExceptionOnScriptError(false);
webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);
webClient.getOptions().setTimeout(10000);
try {
HtmlPage page = webClient.getPage("http://www.bloomberg.com/search?query=Apple");
System.out.println(page.getTitleText());
ScriptResult result = page.executeJavaScript("document.querySelector('"#content > div > section > section.search-results__content > section.content-stories > div.search-result-items > div:nth-child(1) > article > div > h1 > a'").click()");
page = (HtmlPage)result.getNewPage();
System.out.println(page.getTitleText());
} catch (Exception e) {
e.printStackTrace();
} finally {
webClient.close();
}
由于页面不是使用javascript填充的，您也可以完全跳过HtmlUnit并使用像jsoup:
这样的html解析器。新闻类class News{
private String title;
private String href;
private String content="";
public String getTitle() {
return title;
}
public String getHref() {
return href;
}
public String getContent() {
return content;
}
public void setContent(String content) {
this.content = content;
}
public News(String title, String href){
this.title=title;
this.href=href;
}
}
从前两页抓取新闻的示例代码(可通过numberOfResultpages调整):
List<News> bloombergNewsAll = new ArrayList<>();
String searchString = "Apple";
String searchUrl = "http://www.bloomberg.com/search?query=" + searchString + "&page=";
int numberOfResultpages = 2;
Document doc;
// grab title and href
for (int i = 1; i <= numberOfResultpages; i++) {
try {
doc = Jsoup.connect(searchUrl + i)
.userAgent("Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.82 Safari/537.36")
.referrer("http://www.bloomberg.com/").get();
Elements searchResults = doc.select("#content > div > section > section.search-results__content > section.content-stories > div.search-result-items > div > article > div > h1");
if(searchResults.isEmpty()) break; // no more searchResults
for (Element result : searchResults) {
bloombergNewsAll.add(new News(result.text(), result.select("a").attr("href")));
}
} catch (IOException e) {
e.printStackTrace();
}
}
// grab content
for (News news : bloombergNewsAll) {
try {
doc = Jsoup.connect(news.href)
.userAgent("Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.82 Safari/537.36")
.referrer("http://www.bloomberg.com/search?query=Apple").get();
if(news.getHref().contains("bloomberg.com/news/videos")) continue;
if(news.getHref().contains("bloomberg.com/news/")){
news.setContent(doc.select("div.article-body__content").text());
}else if(news.getHref().contains("bloomberg.com/gadfly")){
news.setContent(doc.select("#article > div.body_ZtDFu > div.container_1KxJx").text());
}else if(news.getHref().contains("bloomberg.com/view")){
news.setContent(doc.select("div._31WvjDF17ltgFb1fNB1WqY").text());
}
} catch (IOException e) {
e.printStackTrace();
}
}
// do something useful with your results
for (News news : bloombergNewsAll) {
System.out.println(news.getTitle() + "'n't" + news.getHref() + "'n't" + (news.getContent().length()>150 ? news.getContent().substring(0, 150) : news.getContent()));
}
相关文章：
- 如果我在javascript中输入无效的电子邮件或空白，如何显示特定的文本框边框红色
- jquery代码在Mozilla中有效，但在其他浏览器上无效
- AJAX调用中的非法调用
- 无法在现有状态转换期间更新-未使用任何非法的setState(）
- 这个问号在Flow中意味着什么:“;?(）=>“无效”;
- 从字符串末尾删除空白无效
- 如何解决这种情况下的非法调用类型错误
- Jquery Datatables错误:无效的JSON基元:draw
- 滚动在Chrome中有效，但在Firefox或IE中无效
- 错误:语法错误，无法识别的表达式:不支持的伪:无效/RectJS/Unit Testing
- Javascript setattribute-名称和值无效
- 格式化货币无效
- JavaScript错误无效的限定符
- 按键事件无效的原因
- 在实现基本Sharethis代码时指定了无效或非法的字符串
- WebSocket 语法错误:指定了无效或非法的字符串
- gap.client.load未调用回调:在console.log中指定了一个无效或非法的字符串
- htmlunit:指定了一个无效或非法的选择器
- 模拟页面点击在HtmlUnit(2.33)给出无效或非法的选择器异常
- Blob的DataURI错误:DOM异常5:指定了无效或非法的字符
