python-scraping-notes

my tips about web scraping using python

good to read

useful tools

• chrome plugin SelectorGadget, get the xpath easy

  useful python modueles

tips

file IO

Create a new file

def write_file(path, data,mode='a'):
    with open(path, mode,encoding='utf8') as f:
        f.write(data)

### Delete the contents of a file
def delete_file_contents(path):
    open(path, 'w').close()

get elements

import lxml.html
dom = lxml.html.fromstring(src)

def get_info_xpath( dom, xpath):
    result=""
    for div in dom.xpath(xpath):

        try:
            result = result +div.text +"\n"
            # print(div.text)
        except:

            result = result + div.text_content() + "\n"
            # print(div.text_content())
    return result

useful code block

• manipulate dir and file Path

navigate with css selector,and conditons

for link in soup.select('div a'):
    if(link.find_all("span",class_="cccc")):
        print(link.get('href'))

google translator free API

from googletrans import Translator
trans = Translator(service_urls=[
      'translate.google.com',
      'translate.google.co.jp',
    ])
result = trans.translate(text, dest='zh-CN')
result.text # final text result

-