Request a comparison:
GET: Use GET to request a path to get the returned response. Request parameters will be exposed.HEAD: Especially suitable for priority speed and bandwidth
1. Check the validity of resources.
2. Check the validity of hyperlinks.
3. Check whether the web page has been tampered with.
4. Mostly used for automatic search robots to obtain web page logo information, obtain RSS seed information, or pass security authentication information, etc.
Idea: Allow users to call the specified web dictionary for scanning and add filtering information.
When the status code of the returned URL is 200 and there is no keyword in the filter list, the path is determined to exist.
Code:
<span>import</span> requests<br><span>import</span> os<br><br>urlw=[]<br>okurl=[]<br>noturl=[]<br>error=[<span>'404'</span>,<span>'Not Found'</span>,<span>'403'</span>,<span>'Not Found'</span>,<span>'No Permission'</span>,<span>'360'</span>,<span>'Cloud Lock'</span>,<span>'Website Firewall'</span>,<span>'D Shield'</span>,<span>'Baidu Cloud Acceleration'</span>] <span>//</span>Filter list<br>headers={<span>'user-agent'</span>:<span>'Opera/9.80(WindowsNT6.1;U;en)Presto/2.8. 131Version/11.11'</span>}<br>ld=os.listdir(<span>'dict'</span>)<br><span>print</span>(<span>'[+]Discovery directory'</span>)<br><span>for</span> l <span>in</span> ld:<br> <span>print</span>(<span>'[+]find'</span>,l)<br><br><span>print</span>(<span>''</span>)<br>user=input(<span>'Please choose:'</span>)<br>user2=input(<span>'Enter your URL:'</span>)<br><span>if</span> os.path.exists(<span>'dict/{}'</span>.format(user)):<br> <span>print</span>(<span>'[+]File existence {}'</span>.format(user))<br>else:<br> <span>print</span>(<span>'[-]file does not exist {}'</span>.format(user))<br> exit()<br><br>ope=open(<span>'dict/{}'</span>.format(user),<span>'r'</span>)<br><span>for</span> r <span>in</span> ope.readlines():<br> url=user2.strip()+<span>""</span>.join(r.split(<span>'\n'</span>))<br> urlw.append(url)<br><br>def ether(urls):<br> try:<br> reqt=requests.head(url=urls,headers=headers,allow_redirects=True)<br> <span>for</span> e <span>in</span> error:<br> <span>if</span> reqt.status_code==<span>200</span> <span>and</span> e <span>not</span> <span>in</span> reqt.text: <span>//</span>Judgement<br> <span>yes</span>=<span>'[200]=>Discovery path:{}'</span>.format(reqt.url)<br> <span>if</span> <span>yes</span> <span>in</span> okurl:<span>continue</span> <span>//</span>Remove duplicates<br> okurl.append(<span>yes</span>)<br> <span>print</span>(<span>yes</span>)<br><br> else:<br> <span>no</span >=<span>'[{}]=>Can t find:{}'</span>.format(reqt.status_code,reqt.url)<br> <span>if</span> <span>no</span> <span>in</span> noturl:<span>continue</span> <span>//</span>Remove duplicates<br> noturl.append(<span>no</ span>)<br> <span>print</span>(<span>no</span>)<br> except Exception <span>as</span> g:<br> <span>print</span>(<span>'[-]Error in {} url:{}'</span>.format(g,reqt.url))<br><br><span>for</span> w <span>in</span> urlw:<br> ether(w )<br>
Test results: