1. A sharply divided America decides between Trump and Harris
2. Control of Congress is at stake and with it a president’s agenda
3. Economy ranked as a top issue, but concerns over democracy drove many voters to polls
4. Trump snaps at reporter when asked about abortion: ‘Stop talking about that’
5. Abortion is on the ballot in nine states and motivating voters across the US
6. Election 2024 LIVE: AP coverage and latest news on the presidential election
7. Harris and Trump tie in first election result in Dixville Notch
8. Trump says Republicans showing up at polls in force
9. Here’s what to watch on Election Day in the US
10. Congressional election updates: Senate and House control could be decided by handful of races
11. Voting period marked by massive early turnout ends with few problems, generally smooth Election Day
12. See when polls close in battleground states
13. On the night before Election Day, Kamala Harris brings in celebrities. Donald Trump is unimpressed
14. Trump called Megyn Kelly ‘nasty’ 9 years ago. She just helped deliver his closing message
15. Bernie Sanders seeks a fourth Senate term representing Vermont
16. A presidential campaign unlike any other ends on Tuesday. Here’s how we got here
17. Biden suggests he’d like to smack ‘macho guys’ during final campaign stop
18. Biden declares major disaster area in southeast New Mexico due to historic flooding
19. Cyprus is committed to expanding defense ties with the US, says its president
20. Venezuela’s rejection from developing economies alliance heightens tensions with member Brazil
21. What it’s like in a ‘chaos’ Congress and why these lawmakers keep coming back
22. Jim Jordan failed to become speaker last year. But his rise in the GOP may not be over yet
23. Supreme Court will weigh in on new mostly Black Louisiana congressional district, after election
24. Supreme Court’s conservative justices leave in place Virginia’s purge of voter registrations
25. Virginia asks US Supreme Court to reinstate removals of 1,600 voter registrations
26. A Supreme Court reshaped by Trump has a low profile in this presidential campaign
27. Empty seats become a more common sight at Trump’s final rallies
28. Harris campaign spends final hours reminding Pennsylvania of a Trump ally’s joke about Puerto Rico
29. ‘Fat Leonard,’ Navy contractor behind one of the military’s biggest scandals, sentenced to 15 years
30. Democratic-backed justices look to defend control of Michigan’s Supreme Court
31. Central Michigan voters are deciding 2 open congressional seats in the fight for the US House
32. Beyoncé channels Pamela Anderson in ‘Baywatch’ for Halloween video asking viewers to vote
33. Home Depot’s co-founder and billionaire philanthropist dies at 95
34. Rudy Giuliani ordered to appear in court after missing deadline to turn over Mercedes, other assets
35. Election Day voting underway as Americans chose between Harris or Trump
36. Ground Game: America arrives at Election Day with control of Congress at stake
37. North Dakota measures would end local property taxes and legalize recreational marijuana
38. GOP senator from North Dakota faces Democratic challenger making her 2nd US Senate bid
39. North Dakota’s lone congressman seeks to continue GOP’s decades-old grip on the governor’s post
40. South Dakota is deciding whether to protect abortion rights and legalize recreational marijuana
41. Democratic Rep. Angie Craig seeks a 4th term in Minnesota’s tightest congressional race
42. Democratic Sen. Amy Klobuchar is a heavy favorite to win 4th term against ex-NBA player Royce White
43. Democrats defend Michigan’s open Senate seat, a rare opportunity for Republicans
44. Alaska voters deciding a hard-fought race for the state’s only US House seat, election issues
45. West Virginians’ governor choices stand on opposite sides of the abortion debate
46. A former Trump aide and a longtime congressman are likely to win in high-profile Georgia races
47. GOP Gov. Jim Justice battles Democrat Glenn Elliott for US Senate seat from West Virginia
48. Georgia Democratic prosecutor pursuing election case against Trump faces Republican challenger
49. Voters in battleground Arizona to decide if local agencies can police illegal immigration
50. US Rep. John Curtis is favored to win Mitt Romney’s open Senate seat in Utah
51. Utah Gov. Spencer Cox is expected to win reelection after his surprising endorsement of Trump
52. Democrat Ruben Gallego faces Republican Kari Lake in US Senate race in Arizona
53. Democrats hope to keep winning streak alive in Washington governor’s race
54. GOP tries to break Connecticut Democrats’ winning streak in US House races
55. Landmark Washington climate law faces possible repeal by voters
56. First-term Democrat tries to hold on in Washington state district won by Trump in 2020
57. Justices who split on an abortion measure ruling vie to lead Arkansas Supreme Court
58. Connecticut to decide on constitution change to make mail-in voting easier
ps: как получать cf_clearance надеюсь не надо расcказывать
Здравствуйте, kov_serg, Вы писали: _>Это защита от ботов. Она привязана к ip и user-agent и доказывает прохождении капчи. Для начала можно просто из браузера скопировать, ну а потом:
Понятно. Много дней потратили на изучение получения таких куков? Я умножу и получу результат для себя.
Здравствуйте, Passerby, Вы писали: P>Здравствуйте, kov_serg, Вы писали: _>>Это защита от ботов. Она привязана к ip и user-agent и доказывает прохождении капчи. Для начала можно просто из браузера скопировать, ну а потом: P>Понятно. Много дней потратили на изучение получения таких куков? Я умножу и получу результат для себя.
from __future__ import annotations
import argparse
import asyncio
import io
import json
import logging
import sys
from datetime import datetime, timedelta, timezone
from enum import Enum
from typing import Any, Iterable, List, Optional
import nest_asyncio
import nodriver
from nodriver import cdp
from nodriver.cdp.network import Cookie
from nodriver.core.element import Element
from nodriver.core.tab import Tab
from selenium_authenticated_proxy import SeleniumAuthenticatedProxy
class PrintLocker:
"""A class for locking and unlocking the print function."""def __enter__(self) -> None:
self.unlock()
def __exit__(self, *_: Any) -> None:
self.lock()
@staticmethod
def lock() -> None:
"""Lock the print function."""
sys.stdout = io.StringIO()
@staticmethod
def unlock() -> None:
"""Unlock the print function."""
sys.stdout = sys.__stdout__
class NodriverOptions(list):
"""A class for managing nodriver options."""def add_argument(self, arg: str) -> None:
"""
Add an argument to the list of arguments.
Parameters
----------
arg : str
The argument
"""
self.append(arg)
class ChallengePlatform(Enum):
"""Cloudflare challenge platform types."""
JAVASCRIPT = "non-interactive"
MANAGED = "managed"
INTERACTIVE = "interactive"class CloudflareSolver:
"""
A class for solving Cloudflare challenges with undetected-chromedriver.
Parameters
----------
timeout : float
The timeout in seconds to use for browser actions and solving challenges.
http2 : bool
Enable or disable the usage of HTTP/2 for the browser requests.
headless : bool
Enable or disable headless mode for the browser.
proxy : Optional[str]
The proxy server URL to use for the browser requests.
"""def __init__(
self,
*,
timeout: float,
http2: bool,
headless: bool,
proxy: Optional[str],
) -> None:
options = NodriverOptions()
if not http2:
options.add_argument("--disable-http2")
if headless:
options.add_argument("--headless=new")
if proxy is not None:
auth_proxy = SeleniumAuthenticatedProxy(proxy, use_legacy_extension=True)
auth_proxy.enrich_chrome_options(options)
config = nodriver.Config(browser_args=options)
self.driver = nodriver.Browser(config)
self._timeout = timeout
async def __aenter__(self) -> CloudflareSolver:
await self.driver.start()
return self
async def __aexit__(self, *_: Any) -> None:
self.driver.stop()
@staticmethod
def set_user_agent(tab: Tab, user_agent: str) -> None:
"""
Set the user agent for the browser tab.
Parameters
----------
tab : Tab
The browser tab.
user_agent : str
The user agent string.
"""
tab.feed_cdp(cdp.emulation.set_user_agent_override(user_agent))
@staticmethod
def extract_clearance_cookie(
cookies: Iterable[Cookie],
) -> Optional[Cookie]:
"""
Extract the Cloudflare clearance cookie from a list of cookies.
Parameters
----------
cookies : Iterable[Cookie]
List of cookies.
Returns
-------
Optional[Cookie]
The Cloudflare clearance cookie. Returns None if the cookie is not found.
"""for cookie in cookies:
if cookie.name == "cf_clearance":
return cookie
return None
async def get_cookies(self) -> List[Cookie]:
"""
Get all cookies from the current page.
Returns
-------
List[Cookie]
List of cookies.
"""return await self.driver.cookies.get_all()
async def detect_challenge(self) -> Optional[ChallengePlatform]:
"""
Detect the Cloudflare challenge platform on the current page.
Returns
-------
Optional[ChallengePlatform]
The Cloudflare challenge platform.
"""
html: str = await self.driver.main_tab.get_content()
for platform in ChallengePlatform:
if f"cType: '{platform.value}'"in html:
return platform
return None
async def solve_challenge(self) -> None:
"""Solve the Cloudflare challenge on the current page."""
start_timestamp = datetime.now()
while (
self.extract_clearance_cookie(await self.get_cookies()) is None
and await self.detect_challenge() is not None
and (datetime.now() - start_timestamp).seconds < self._timeout
):
widget_input = await self.driver.main_tab.find("input")
if widget_input.parent is None or not widget_input.parent.shadow_roots:
await asyncio.sleep(0.25)
continue
challenge = Element(
widget_input.parent.shadow_roots[0],
self.driver.main_tab,
widget_input.parent.tree,
)
challenge = challenge.children[0]
if (
isinstance(challenge, Element)
and"display: none;"not in challenge.attrs["style"]
):
await asyncio.sleep(1)
try:
await challenge.get_position()
except Exception:
continue
await challenge.mouse_click()
async def main() -> None:
parser = argparse.ArgumentParser(
description="A simple program for scraping Cloudflare clearance (cf_clearance) cookies from websites issuing Cloudflare challenges to visitors"
)
parser.add_argument(
"url",
metavar="URL",
help="The URL to scrape the Cloudflare clearance cookie from",
type=str,
)
parser.add_argument(
"-f",
"--file",
default=None,
help="The file to write the Cloudflare clearance cookie information to, in JSON format",
type=str,
)
parser.add_argument(
"-t",
"--timeout",
default=30,
help="The timeout in seconds to use for solving challenges",
type=float,
)
parser.add_argument(
"-p",
"--proxy",
default=None,
help="The proxy server URL to use for the browser requests",
type=str,
)
parser.add_argument(
"-ua",
"--user-agent",
default="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36",
help="The user agent to use for the browser requests",
type=str,
)
parser.add_argument(
"--disable-http2",
action="store_true",
help="Disable the usage of HTTP/2 for the browser requests",
)
parser.add_argument(
"-d",
"--debug",
action="store_true",
help="Run the browser in headed mode",
)
parser.add_argument(
"-v",
"--verbose",
action="store_true",
help="Increase the output verbosity",
)
args = parser.parse_args()
nest_asyncio.apply()
print_locker = PrintLocker()
print_locker.lock()
logging_level = logging.INFO if args.verbose else logging.ERROR
logging.basicConfig(
format="[%(asctime)s] [%(levelname)s] %(message)s",
datefmt="%H:%M:%S",
level=logging_level,
)
logging.getLogger("nodriver").setLevel(logging.WARNING)
logging.info("Launching %s browser...", "headed"if args.debug else"headless")
challenge_messages = {
ChallengePlatform.JAVASCRIPT: "Solving Cloudflare challenge [JavaScript]...",
ChallengePlatform.MANAGED: "Solving Cloudflare challenge [Managed]...",
ChallengePlatform.INTERACTIVE: "Solving Cloudflare challenge [Interactive]...",
}
async with CloudflareSolver(
timeout=args.timeout,
http2=not args.disable_http2,
headless=not args.debug,
proxy=args.proxy,
) as solver:
solver.set_user_agent(solver.driver.main_tab, args.user_agent)
# fix.begin
ua,_=await solver.driver.main_tab.send(cdp.runtime.evaluate(expression="navigator.userAgent"))
if args.user_agent!=ua.value:
args.user_agent=ua.value
logging.info("user_agent=%s",ua.value)
# fix.end
await solver.driver.main_tab.reload()
logging.info("Going to %s...", args.url)
try:
await solver.driver.get(args.url)
except asyncio.TimeoutError as err:
logging.error(err)
return
clearance_cookie = solver.extract_clearance_cookie(await solver.get_cookies())
if clearance_cookie is not None:
logging.info("Cookie: cf_clearance=%s", clearance_cookie.value)
logging.info("User agent: %s", args.user_agent)
if not args.verbose:
with print_locker:
print(f"cf_clearance={clearance_cookie.value}")
return
challenge_platform = await solver.detect_challenge()
if challenge_platform is None:
logging.error("No Cloudflare challenge detected.")
return
logging.info(challenge_messages[challenge_platform])
try:
await solver.solve_challenge()
except asyncio.TimeoutError:
pass
clearance_cookie = solver.extract_clearance_cookie(await solver.get_cookies())
if clearance_cookie is None:
logging.error("Failed to retrieve a Cloudflare clearance cookie.")
return
logging.info("Cookie: cf_clearance=%s", clearance_cookie.value)
logging.info("User agent: %s", args.user_agent)
if not args.verbose:
with print_locker:
print(f"cf_clearance={clearance_cookie.value}")
if args.file is None:
return
logging.info("Writing Cloudflare clearance cookie information to %s...", args.file)
try:
with open(args.file, encoding="utf-8") as file:
json_data = json.load(file)
except (FileNotFoundError, json.JSONDecodeError):
json_data = {"clearance_cookies": []}
local_timezone = datetime.now(timezone.utc).astimezone().tzinfo
unix_timestamp = clearance_cookie.expires - timedelta(days=365).total_seconds()
timestamp = datetime.fromtimestamp(unix_timestamp, tz=local_timezone).isoformat()
json_data["clearance_cookies"].append(
{
"unix_timestamp": int(unix_timestamp),
"timestamp": timestamp,
"domain": clearance_cookie.domain,
"cf_clearance": clearance_cookie.value,
"user_agent": args.user_agent,
"proxy": args.proxy,
}
)
with open(args.file, "w", encoding="utf-8") as file:
json.dump(json_data, file, indent=4)
if __name__ == "__main__":
asyncio.run(main())
output
[16:27:56] [INFO] Launching headed browser...
[16:27:56] [INFO] Going to https://apnews.com/...
[16:27:57] [INFO] Solving Cloudflare challenge [Managed]...
[16:28:00] [INFO] Cookie: cf_clearance=GB1G2gzWCizNO_BrRHcHi_Y1tvA3SXlayBABwc6fzuM-1730899677-1.2.1.1-0feVnS_aJeeUko7dqVUKdw.8AekwrxvYhS5shs3tj2ecGdvb8HEVZVLg4euN2CpJJw5TxxITnb7El89dMeVWg7YylOC_bAcg3Byo1_CfCXOa_zeo2LKuh1hPMe8IFGoqLDfQm5HshstBd67q6spGO.9hN6RfD9jMXswhoD9yZjKhIgbfunaKuGnJzv2gBk_LtDg9N9dFWFI6SlowCLUDsenyxKVdZgIQjZUsr_.f7TKj7CxX2uR8PBCHre2GIgF8P63HChfNJ5wqKiM4MzIWYHWFnBFtRq3Joe.4IkCj6r7F7nsUErH518kOU8wrZQ_OBny3Pa6E.9icq8J.Ta7p8kGbp7prgawJ.Cdp4jV.wgpXNEyUP0l3xXk8gHL0OuV3b_Lg5_vPO5FagD.3ZohRtBsMMN8u0ZF1ekW3zWLA09g
[16:28:00] [INFO] User agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36
[16:28:00] [INFO] Writing Cloudflare clearance cookie information to config.json...
[16:28:01] [WARNING] Loop <_UnixSelectorEventLoop running=False closed=True debug=False> that handles pid 51860 is closed
1. Oregon gets top billing in College Football Playoff’s opening rankings, Ohio St 2nd and Georgia 3rd
2. Dodgers star Shohei Ohtani has surgery to repair labrum tear in shoulder after World Series injury
3. NFL trade deadline: Commanders acquire Lattimore; Lions get Za’Darius Smith; Steelers add Williams
4. 76ers’ Joel Embiid is suspended by the NBA for three games for shoving a newspaper columnist
5. Oklahoma State coach Mike Gundy apologizes after lashing out at critics
6. Fans celebrate as Los Angeles Dodgers win World Series
7. Michael Jordan speaks after NASCAR court hearing
8. Japan celebrates Shohei Ohtani as Dodgers win World Series
9. U.S. Nordic combined salvages program with grant from International Ski and Snowboard Federation
10. Real Madrid, Manchester City both humiliated in Champions League, Liverpool enjoys Alonso’s return
11. Jerry Jones says Dak Prescott is likely headed to IR, but owner isn’t giving up on Cowboys’ season
12. The Commanders acquire 4-time Pro Bowl cornerback Marshon Lattimore from the Saints
13. Clemson coach Dabo Swinney challenged at poll when out to vote in election
14. Which move is better down 1 late? Kick the extra point or go for 2
15. Bills release safety Mike Edwards after failing in a bid to trade him before NFL deadline
16. Ravens QB Lamar Jackson misses practice again, but his coach says he’ll play against Cincinnati
17. Bowles says the Bucs need to do little things better to pull out of midseason tailspin
18. Florida finds its footing under coach Billy Napier. It should be enough to earn him a 4th year
19. From rankings, to 4-wheelers, to tortillas, Deion Sanders had plenty on his mind after the bye week
20. South Carolina off to another late surge after convincing win over Top 10 Texas A&M
21. Preseason Big 12 top-five teams Utah, Oklahoma St., Kansas and Arizona at bottom of league standings
22. Spurs forward Sochan having surgery to repair fractured left thumb suffered in loss to Clippers
23. Hornets lose center Nick Richards for at least two weeks with rib cartilage fracture
24. Nuggets forward Aaron Gordon expected to miss multiple weeks with right calf strain, AP sources say
25. Clippers overcome 26-point deficit for their 1st victory at Intuit Dome
26. Celebrini returns from injury and Wennberg scores in OT to lift Sharks over Blue Jackets 2-1
27. Quinn Hughes and Brock Boeser score big in Canucks’ 5-1 rout of Ducks
28. Lehkonen scores go-ahead goal in return from injury as Avalanche beat Kraken 6-3
29. Lewis scores twice, Kings roll to 5-1 win over Wild
30. Led by Mark Sears, No. 2 Alabama is perhaps SEC’s best hope to end national title drought
31. Sixth-ranked Gonzaga opens with 101-63 victory over No. 8 Baylor
32. Unranked Ohio State leads wire-to-wire in 80-72 victory over No. 19 Texas
33. Love scores 17 and No. 10 Arizona opens season with 93-64 win over Canisius
34. South Carolina forward Ashlyn Watkins has charges against her dismissed
35. No. 1 South Carolina avoids major upset in 68-62 win over Michigan
36. No. 5 UCLA women overcome slow start to beat No. 17 Louisville 66-59 in Paris
37. No. 3 USC hits two late free throws to beat Ole Miss 68-66 in Paris
38. OMG hits Cooperstown as Mets’ home run sign will get Hall of Fame display
39. Braves stars Ronald Acuña Jr. and Spencer Strider not expected back from injuries by 2025 opener
40. Yankees GM Brian Cashman says he’s talked with agent Scott Boras about Juan Soto and Pete Alonso
41. Cashman sounds as if he intends to bring back Boone as manager, defends Yankees from Kelly criticism
42. Ancelotti ‘worried’ after another poor performance by Real Madrid
43. Arne Slot humbles Xabi Alonso as Liverpool routs Bayer Leverkusen 4-0 in the Champions League
44. Liverpool forward Luis Diaz scores second-half hat trick in 4-0 rout of Bayer Leverkusen
45. Amorim heading into Man United job on back of huge win over Man City
46. Judge rules Benjamin Mendy entitled to majority of his claim against former club Man City
47. McIlroy looks to clinch Race to Dubai title with new swing after 3 weeks shut away in a studio
48. Sauber signs Brazilian driver Gabriel Bortoleto to partner Nico Hülkenberg in F1 for 2025
49. South Dakota Coyotes play the Texas A&M-Commerce Lions in non-conference action
50. Le Moyne and CSU Northridge set for cross-conference contest
51. No. 17 Indiana takes on SIU-Edwardsville for cross-conference game
52. Jacksonville State visits Air Force after Taylor’s 30-point outing
53. White leads North Dakota State against Illinois State after 22-point game
54. Central Michigan Chippewas host the Stony Brook Seawolves in non-conference action
55. Alcorn State Braves head to the Utah State Aggies
56. Jacksonville visits No. 21 Florida following Clayton’s 29-point game
57. West Georgia set for road matchup with the Georgia Tech Yellow Jackets
58. Olowoniyi leads Southern Indiana against Bucknell after 23-point outing
59. Cato leads Central Arkansas against Utah after 21-point game
60. Idaho State visits USC for non-conference showdown
61. Southern Miss visits UAB after Lendeborg’s 22-point game
62. Idaho hosts UC Davis following Johnson’s 35-point game
63. LSU hosts UL Monroe in out-of-conference matchup
64. Villanova takes on Columbia after Poplar’s 20-point showing
65. Florida A&M visits SMU after Miller’s 21-point game
66. Southern faces Iowa after Dioumassi’s 30-point game
67. High Point hosts Coppin State for cross-conference contest
68. Chattanooga heads to Saint Mary’s (CA) for non-conference matchup
69. Boise State takes on Oakland in non-conference matchup
70. Pennsylvania hosts Maryland-Eastern Shore after Shaw’s 23-point game
71. No. 15 Creighton Bluejays to host UT Rio Grande Valley Vaqueros Wednesday
72. North Florida hosts Charleston Southern in cross-conference matchup
73. No. 25 Rutgers Scarlet Knights open season at home against the Wagner Seahawks
74. Northern Iowa Panthers to take on the Milwaukee Panthers Thursday
75. Stetson Hatters to host the Omaha Mavericks on Thursday
using System;
using System.IO;
using System.Net;
using System.Text.Json;
using HtmlAgilityPack;
class Program
{
static void Main()
{
// string url = "https://apnews.com/politics";string url = "https://apnews.com/sports";
string userAgent = "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36";
string cfgFn = "config.json";
if (!File.Exists(cfgFn))
{
var processInfo = new System.Diagnostics.ProcessStartInfo
{
FileName = "python3",
Arguments = $"main.py https://apnews.com/ -v -d -f {cfgFn} --user-agent='{userAgent}'",
RedirectStandardOutput = true,
UseShellExecute = false,
CreateNoWindow = true
};
var process = System.Diagnostics.Process.Start(processInfo);
process.WaitForExit();
}
var cfg = JsonSerializer.Deserialize<Config>(File.ReadAllText(cfgFn));
userAgent = cfg.ClearanceCookies[0].UserAgent;
string cfClearance = cfg.ClearanceCookies[0].CfClearance;
var request = (HttpWebRequest)WebRequest.Create(url);
request.Method = "GET";
request.UserAgent = userAgent;
request.Headers.Add("Cookie", $"cf_clearance={cfClearance}");
string htmlContent;
using (var response = (HttpWebResponse)request.GetResponse())
using (var reader = new StreamReader(response.GetResponseStream()))
{
htmlContent = reader.ReadToEnd();
}
var htmlDoc = new HtmlDocument();
htmlDoc.LoadHtml(htmlContent);
var titles = htmlDoc.DocumentNode.SelectNodes("//div[contains(@class, 'PagePromo-title')]");
if (titles == null)
{
File.Delete(cfgFn);
Console.WriteLine("fail");
return;
}
int i = 0;
foreach (var title in titles)
{
string newsText = title.InnerText.Trim();
string newsUrl = string.Empty;
var link = title.SelectSingleNode(".//a");
if (link != null) newsUrl = link.GetAttributeValue("href", string.Empty);
Console.WriteLine($"{++i,2}. {newsText}");
}
}
public class Config
{
public ClearanceCookie[] ClearanceCookies { get; set; }
}
public class ClearanceCookie
{
public string UserAgent { get; set; }
public string CfClearance { get; set; }
}
}
На строке var cfg = JsonSerializer.Deserialize<Config>(File.ReadAllText(cfgFn)); ошибка во время выполнения: пишет, что нет файла config.json. Совсем разработчики PHP to C# обленились, не могут нормальный код выдать)). Как бы исправить?)
P>На строке var cfg = JsonSerializer.Deserialize<Config>(File.ReadAllText(cfgFn)); ошибка во время выполнения: пишет, что нет файла config.json. Совсем разработчики PHP to C# обленились, не могут нормальный код выдать)). Как бы исправить?)
Сначала с командной строки вручную запустите полчучение cf_clearance
(должен быть chrome и зависимости pip install -r requirements.txt)
ps: в винде скорее всего надо писать просто python. (нужен python3.9+)
pps: там еще какая-то бага которая не даёт менять user_agent и в результате в режиме headless передаётся что оно headless и проверка пролетает. поэтому там ключ -d и [fix.begin .. fix.end]
Здравствуйте, Passerby, Вы писали:
P>Здравствуйте, kov_serg, Вы писали:
P>Успешно установил: P>pip install cf_clearance P>pip install nest_asyncio P>pip install nodriver P>pip install selenium_authenticated_proxy
P>Но: P>python main.py https://apnews.com/ -v -d -f config.json P>[17:20:32] [INFO] Launching headed browser... P>[17:20:33] [INFO] user_agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36 P>[17:20:33] [INFO] Going to https://apnews.com/... P>[17:20:33] [ERROR] No Cloudflare challenge detected.
P>Хотя здесь https://github.com/Xewdy444/CF-Clearance-Scraper/issues/24 и написано, что исправлено. Но, увы..
Посмотрите что у браузера такой же user_agent как указан "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"
Если нет поменяйте его.
Здравствуйте, kov_serg, Вы писали: _>Посмотрите что у браузера такой же user_agent как указан "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" _>Если нет поменяйте его.
Посмотрел и исправил:
user-agent:
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36
Т.е. не Chrome/130, а Chrome/131.
Снова запустил
python main.py https://apnews.com/ -v -d -f config.json
[05:25:15] [INFO] Launching headed browser...
[05:25:16] [INFO] Going to https://apnews.com/...
[05:25:17] [ERROR] No Cloudflare challenge detected.
Т.е. теперь нет второй строчки user_agent=..., но все равно [ERROR].
Здравствуйте, kov_serg, Вы писали: _>Вы просто зайдите на https://apnews.com/ _>он теперь не проводит проверку cloudflare можно просто выполнять запросы без заморочек c cf_clearance
using OpenQA.Selenium.Chrome;
using System.Text;
class Program
{
static void Main(string[] args)
{
var chromeOptions = new ChromeOptions();
var driver = new ChromeDriver(chromeOptions);
// open the target page in Chrome
driver.Navigate().GoToUrl("https://www.cryptocompare.com/news/list/latest/");
var html = driver.PageSource;
//парсинг
// close the browser and release its resources
driver.Quit();
}
}
01. Coinbase To Add Support For Parcl (PRCL) On Solana Network
02. Chainlink and Hedera Unite To Revolutionize DeFi and Real-World Asset Tokenization
03. Trump’s Bitcoin Reserve (TRUMPRES) Solana Memecoin to Rally 17,000% Before Exchange Listing, As SHIB and DOGE Lag
04. UK FCA FUDs Retardio meme coin
05. UK FCA FUDs Retardio meme coin
06. FTX Plans to Start Customer Repayments by Early 2025 as Bankruptcy Process Advances
07. 4 Ethereum Rivals Altcoins That Could Double Before January 2025
08. Crypto Funds Attract Record $20.3 Billion in Inflows Over 10 Weeks
09. FTX Will Repay Customers in Early 2025 via Kraken and BitGo
10. UK FCA Flags Retardio Solana For Unregulated Financial Services Promotion
11. Surge in Xaman User Activity Suggests Renewed Interest in XRP and Anticipation for RLUSD Stablecoin Launch
12. Bitcoin’s DeFi TVL Surges Over Sixfold Amid Growing Restaking Demand and Institutional Interest
13. Nasdaq-Listed Crypto Firm Expands with Liberty Road Capital Acquisition and Xapo Bank Partnership
14. Bitcoin DeFi TVL hits record highs, flips BNB – Good news for BTC?
15. Pendle price prediction 2024-2030: Is PENDLE a good investment?
16. XRP rally and RLUSD anticipation fuel activity on Xaman Wallet
17. Singapore Leading World in Crypto Adoption, With 40% of Investors Holding Digital Assets: Report
18. 21Shares XRP Trust Officially Registered in Delaware | XRP Price Reaches $2.52
19. FCA releases discussion paper on crypto market transparency, abuse
20. Ethereum Price Today As 92% ETH Holders Are Profitable
21. XRP Price Eyes Rally To $7 As Whales Shuffle $283M
22. BREAKING: Here are Date and Details on FTX Refunds That May Affect the Cryptocurrency Market
23. Coinbase Lists Parcl (PRCL) for Trading
24. Kraken chosen as distribution partner by the FTX Debtors
25. FTX Set to Resume Operations on January 3, 2025: Latest Cryptocurrency News
26. ‘Prepare your exit strategy,’ warns crypto trader as ‘we are close to the top’
27. Bitcoin Hits New All-Time High, Sparking Speculation of Possible Rally Toward $113,000
28. Bitcoin Breaks $107,000 Amid Fed Rate Cut Expectations and Trump’s Crypto Reserve Plans
29. PENGU: Everything You Need to Know About the Pudgy Penguins Solana Token and Airdrop
30. Top Ripple (XRP) Price Predictions as of Late
31. Bitcoin hits new record high with surge to $107k
32. Bitcoin hits new record high with surge to $107k
33. Quiet Powerhouses: Are Ethereum and Cardano Ready for Big Upswings?
34. Ripple’s RLUSD scheduled for Dec. 17 launch
35. Ripple’s RLUSD scheduled for Dec. 17 launch
36. Price analysis 12/16: SPX, DXY, BTC, ETH, XRP, SOL, BNB, DOGE, ADA, AVAX
37. Chainlink price rallies as hopes of an 80% surge rise
38. Chainlink price rallies as hopes of an 80% surge rise
39. Pepe And Mog See Overnight Declines, Analysts Highlight This Project As A High ROI Alternative
40. From Crypto Genius to Centi-Billionaire: Satoshi Nakamoto’s Wealth Soars
41. Time To Sell XRP? Price Completes Head And Shoulder Pattern, Suggesting Crash To $2.2 Is Imminent
42. Pepe Coin Millionaire Identifies the ‘Next PEPE’ That Could Turn $300 Into $30,000 in Just 8 Weeks
43. Semler Scientific buys more Bitcoin, now holds 2,084 BTC
44. Semler Scientific buys more Bitcoin, now holds 2,084 BTC
45. Ripple Launches RLUSD Stablecoin Amid XRP Price Surge and Growing Institutional Interest
46. Arbitrum price prediction 2024 – 2030: ARB breaks key barrier
47. Lido sunsets staking on Polygon network
48. Lido sunsets staking on Polygon network
49. Here’s when Chainlink (LINK) will reach $40, according to analyst
50. UK to Ban Public Crypto Offers in Incisive New Regulatory Climate
Здравствуйте, kov_serg, Вы писали: _>Постораюсь еще раз намекнуть. Там нет проверки cloudflar
А на https://www.yahoo.com/ или на https://www.bloomberg.com/europe она есть? Можете указать сайт, на котором она есть? Хочется запустить код.