Selenium/Webscrape this field

Question 1

我的代碼運行良好，打印所有行的標題，但帶有下拉列表的行除外。

例如，如果單擊，第4行有一個下拉列表。我實現了一個嘗試，在理論上會啟動下拉列表，然后拉標題。

但是當我執行click（）并嘗試打印時，對于帶有這些下拉列表的行，它們不會打印。

應為output-打印所有標題，包括下拉列表中的標題。

from selenium import webdriver
from bs4 import BeautifulSoup
import time
driver = webdriver.Chrome()
driver.get('https://cslide.ctimeetingtech.com/esmo2021/attendee/confcal/session/list')
time.sleep(4)
page_source = driver.page_source
soup = BeautifulSoup(page_source,'html.parser')

productlist=soup.find_all('div',class_='card item-container session')
for property in productlist:
    sessiontitle=property.find('h4',class_='session-title card-title').text
    print(sessiontitle)
    try:
        ifDropdown=driver.find_elements_by_class_name('item-expand-action expand')
        ifDropdown.click()
        time.sleep(4)
        newTitle=driver.find_element_by_class_name('card-title').text
        print(newTitle)
    except:
        newTitle='none'

Question 2

import requests
from bs4 import BeautifulSoup
import pandas as pd


def get_soup(content):
    return BeautifulSoup(content, 'lxml')


def my_filter(req, content):
    try:
        r = req.get(content['href'])
        soup = get_soup(r.text)
        return [x.text for x in soup.select('.card-title')[1:]]
    except TypeError:
        return 'N/A'


def main(url):
    with requests.Session() as req:
        for page in range(1, 2):
            print(f"Extracting Page# {page}\n")
            params = {
                "p": page
            }
            r = req.get(url, params=params)
            soup = get_soup(r.text)
            goal = {x.select_one('.session-title').text: my_filter(
                req, x.select_one('.item-expand-action')) for x in soup.select('.card')}
        df = pd.DataFrame(goal.items(), columns=['Title', 'Menu'])
        print(df)


main('https://cslide.ctimeetingtech.com/esmo2021/attendee/confcal/session/list')

Output:

                                                Title                                               Menu
0                      Educational sessions on-demand                                                N/A
1                          Special Symposia on-demand                                                N/A
2                Multidisciplinary sessions on-demand                                                N/A
3   Illumina - Diagnosing Non-Small Cell Lung Canc...  [Illumina gives an update on their IVD road ma...
4   MSD - Homologous Recombination Deficiency: BRC...  [Welcome and Introductions, Homologous Recombi...
5   Servier - The clinical value of IDH inhibition...  [Isocitric dehydrogenase: an actionable geneti...
6   AstraZeneca - Redefining Breast Cancer – Biolo...  [Welcome and Opening, Redefining Breast Cancer...
7   ITM Isotopen Technologien München AG - A Globa...  [Welcome & Introduction, Changes in the Incide...
8   MSD - The Role of Biomarkers in Patient Manage...  [Welcome and Introductions, The Role of Pd-L1 ...
9   AstraZeneca - Re-evaluating the role of gBRCA ...  [Welcome and introduction, What do we know abo...
10  Novartis - Unmet needs in oncogene-driven NSCL...  [Welcome and introduction, Unmet needs in onco...
11                                    Opening session                                                N/A

Answer 1

import requests
from bs4 import BeautifulSoup
import pandas as pd


def get_soup(content):
    return BeautifulSoup(content, 'lxml')


def my_filter(req, content):
    try:
        r = req.get(content['href'])
        soup = get_soup(r.text)
        return [x.text for x in soup.select('.card-title')[1:]]
    except TypeError:
        return 'N/A'


def main(url):
    with requests.Session() as req:
        for page in range(1, 2):
            print(f"Extracting Page# {page}\n")
            params = {
                "p": page
            }
            r = req.get(url, params=params)
            soup = get_soup(r.text)
            goal = {x.select_one('.session-title').text: my_filter(
                req, x.select_one('.item-expand-action')) for x in soup.select('.card')}
        df = pd.DataFrame(goal.items(), columns=['Title', 'Menu'])
        print(df)


main('https://cslide.ctimeetingtech.com/esmo2021/attendee/confcal/session/list')

Output:

                                                Title                                               Menu
0                      Educational sessions on-demand                                                N/A
1                          Special Symposia on-demand                                                N/A
2                Multidisciplinary sessions on-demand                                                N/A
3   Illumina - Diagnosing Non-Small Cell Lung Canc...  [Illumina gives an update on their IVD road ma...
4   MSD - Homologous Recombination Deficiency: BRC...  [Welcome and Introductions, Homologous Recombi...
5   Servier - The clinical value of IDH inhibition...  [Isocitric dehydrogenase: an actionable geneti...
6   AstraZeneca - Redefining Breast Cancer – Biolo...  [Welcome and Opening, Redefining Breast Cancer...
7   ITM Isotopen Technologien München AG - A Globa...  [Welcome & Introduction, Changes in the Incide...
8   MSD - The Role of Biomarkers in Patient Manage...  [Welcome and Introductions, The Role of Pd-L1 ...
9   AstraZeneca - Re-evaluating the role of gBRCA ...  [Welcome and introduction, What do we know abo...
10  Novartis - Unmet needs in oncogene-driven NSCL...  [Welcome and introduction, Unmet needs in onco...
11                                    Opening session                                                N/A

Selenium/Webscrape this field

熱門問答

當更新一個字段的數據時，顯示對象中另一個字段未定義

如何確定標題標記的值？

無法在PHP中使用openssl_decrypt&AES解密電子郵件地址

汽車租賃系統中具有天窗功能的最后一輛租賃汽車的SQL查詢

我可以將數組的所有元素作為單獨的字符串獲取嗎？

列包含空值

從Facebook令牌元數據響應中獲取'user_id'值

NotImplementedError: Operation not supported or unimplemented on this platform怎么解決

F#與Excel-DNA聯合使用時，如何管理內存和資源

'Sequential' object has no attribute 'compilep'

async的fn寫throw會被promise.catch捕獲到嗎？

執行npm run build: 報錯 internal/child_process.js:323 spawn EBUSY