open.source/whoogle-search
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'origin/main' into heroku-app

Browse Source
This commit is contained in:
Ben Busby 2021-04-05 12:29:08 -04:00
commit ca271d3ce1
No known key found for this signature in database
GPG Key ID: 3B08611DF6E62ED2
50 changed files with 1145 additions and 491 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
.github/ISSUE_TEMPLATE/feature_request.md vendored
View File

@ -7,6 +7,12 @@ assignees: ''
---
<!--
DO NOT REQUEST UI/THEME/GUI/APPEARANCE IMPROVEMENTS HERE
THESE SHOULD GO IN ISSUE #60
REQUESTING A NEW FEATURE SHOULD BE STRICTLY RELATED TO NEW FUNCTIONALITY
-->
**Describe the feature you'd like to see added**
A short description of the feature, and what it would accomplish.

2
.github/workflows/buildx.yml vendored
View File

@ -21,6 +21,8 @@ jobs:
docker login -u "${{ secrets.DOCKER_USERNAME }}" --password-stdin
- name: build and push the image
run: |
docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
docker buildx ls
docker buildx build --push \
--tag benbusby/whoogle-search:buildx-experimental \
--platform linux/amd64,linux/arm/v7,linux/arm64 .

43
Dockerfile
View File

@ -1,18 +1,23 @@
FROM python:3.8-slim
FROM python:3.8-slim as builder
WORKDIR /usr/src/app
RUN apt-get update && apt-get install -y \
build-essential \
libcurl4-openssl-dev \
libssl-dev \
libxml2-dev \
libxslt-dev \
libffi-dev \
tor
libssl-dev \
libffi-dev
COPY config/tor/torrc /etc/tor/torrc
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
RUN pip install --prefix /install --no-warn-script-location --no-cache-dir -r requirements.txt
FROM python:3.8-slim
RUN apt-get update && apt-get install -y \
libcurl4-openssl-dev \
tor \
wget \
&& rm -rf /var/lib/apt/lists/*
ARG config_dir=/config
RUN mkdir -p $config_dir
@ -35,6 +40,10 @@ ENV WHOOGLE_PROXY_TYPE=$proxytype
ARG proxyloc=''
ENV WHOOGLE_PROXY_LOC=$proxyloc
ARG whoogle_dotenv=''
ENV WHOOGLE_DOTENV=$whoogle_dotenv
ARG use_https=1
ENV HTTPS_ONLY=$use_https
ARG whoogle_port=5000
@ -45,10 +54,22 @@ ENV WHOOGLE_ALT_TW=$twitter_alt
ARG youtube_alt='invidious.snopyta.org'
ENV WHOOGLE_ALT_YT=$youtube_alt
ARG instagram_alt='bibliogram.art/u'
ENV WHOOGLE_ALT_YT=$instagram_alt
ENV WHOOGLE_ALT_IG=$instagram_alt
ARG reddit_alt='libredd.it'
ENV WHOOGLE_ALT_RD=$reddit_alt
COPY . .
WORKDIR /whoogle
COPY --from=builder /install /usr/local
COPY misc/tor/torrc /etc/tor/torrc
COPY misc/tor/start-tor.sh misc/tor/start-tor.sh
COPY app/ app/
COPY run .
COPY whoogle.env .
EXPOSE $EXPOSE_PORT
CMD config/tor/start-tor.sh & ./run
HEALTHCHECK --interval=5m --timeout=5s \
CMD wget --no-verbose --tries=1 http://localhost:${EXPOSE_PORT}/ || exit 1
CMD misc/tor/start-tor.sh & ./run

65
README.md
View File

@ -17,7 +17,8 @@ Contents
5. [Usage](#usage)
6. [Extra Steps](#extra-steps)
7. [FAQ](#faq)
8. [Screenshots](#screenshots)
8. [Public Instances](#public-instances)
9. [Screenshots](#screenshots)
## Features
- No ads or sponsored content
@ -55,7 +56,7 @@ If using Heroku Quick Deploy, **you can skip this section**.
There are a few different ways to begin using the app, depending on your preferences:
### A) [Heroku Quick Deploy](https://heroku.com/about)
[![Deploy](https://www.herokucdn.com/deploy/button.svg)](https://heroku.com/deploy?template=https://github.com/benbusby/whoogle-search/tree/heroku-app)
[![Deploy](https://www.herokucdn.com/deploy/button.svg)](https://heroku.com/deploy?template=https://github.com/benbusby/whoogle-search/tree/heroku-app-beta)
*Note: Requires a (free) Heroku account*
@ -136,6 +137,9 @@ Description=Whoogle
#Environment=WHOOGLE_ALT_TW=nitter.net
#Environment=WHOOGLE_ALT_YT=invidious.snopyta.org
#Environment=WHOOGLE_ALT_IG=bibliogram.art/u
#Environment=WHOOGLE_ALT_RD=libredd.it
# Load values from dotenv only
#Environment=WHOOGLE_DOTENV=1
Type=simple
User=root
WorkingDirectory=<whoogle_directory>
@ -218,6 +222,9 @@ heroku open
This series of commands can take a while, but once you run it once, you shouldn't have to run it again. The final command, `heroku open` will launch a tab in your web browser, where you can test out Whoogle and even [set it as your primary search engine](https://github.com/benbusby/whoogle#set-whoogle-as-your-primary-search-engine).
You may also edit environment variables from your app’s Settings tab in the Heroku Dashboard.
#### Arch Linux & Arch-based Distributions
There is an [AUR package available](https://aur.archlinux.org/packages/whoogle-git/), as well as a pre-built and daily updated package available at [Chaotic-AUR](https://chaotic.cx).
#### Using your own server, or alternative container deployment
There are other methods for deploying docker containers that are well outlined in [this article](https://rollout.io/blog/the-shortlist-of-docker-hosting/), but there are too many to describe set up for each here. Generally it should be about the same amount of effort as the Heroku deployment.
@ -228,21 +235,39 @@ Depending on your preferences, you can also deploy the app yourself on your own
- A bit more experience or willingness to work through issues
## Environment Variables
There are a few optional environment variables available for customizing a Whoogle instance:
There are a few optional environment variables available for customizing a Whoogle instance. These can be set manually, or copied into `whoogle.env` and enabled by setting `WHOOGLE_DOTENV=1`.
| Variable | Description |
| ------------------ | -------------------------------------------------------------- |
| WHOOGLE_USER | The username for basic auth. WHOOGLE_PASS must also be set if used. |
| WHOOGLE_PASS | The password for basic auth. WHOOGLE_USER must also be set if used. |
| WHOOGLE_PROXY_USER | The username of the proxy server. |
| WHOOGLE_PROXY_PASS | The password of the proxy server. |
| WHOOGLE_PROXY_TYPE | The type of the proxy server. Can be "socks5", "socks4", or "http". |
| WHOOGLE_PROXY_LOC | The location of the proxy server (host or ip). |
| EXPOSE_PORT | The port where Whoogle will be exposed. |
| Variable | Description |
| ------------------ | ----------------------------------------------------------------------------------------- |
| WHOOGLE_DOTENV | Load environment variables in `whoogle.env` |
| WHOOGLE_USER | The username for basic auth. WHOOGLE_PASS must also be set if used. |
| WHOOGLE_PASS | The password for basic auth. WHOOGLE_USER must also be set if used. |
| WHOOGLE_PROXY_USER | The username of the proxy server. |
| WHOOGLE_PROXY_PASS | The password of the proxy server. |
| WHOOGLE_PROXY_TYPE | The type of the proxy server. Can be "socks5", "socks4", or "http". |
| WHOOGLE_PROXY_LOC | The location of the proxy server (host or ip). |
| EXPOSE_PORT | The port where Whoogle will be exposed. |
| HTTPS_ONLY | Enforce HTTPS. (See [here](https://github.com/benbusby/whoogle-search#https-enforcement)) |
| WHOOGLE_ALT_TW | The twitter.com alternative to use when site alternatives are enabled in the config. |
| WHOOGLE_ALT_YT | The youtube.com alternative to use when site alternatives are enabled in the config. |
| WHOOGLE_ALT_IG | The instagram.com alternative to use when site alternatives are enabled in the config. |
| WHOOGLE_ALT_TW | The twitter.com alternative to use when site alternatives are enabled in the config. |
| WHOOGLE_ALT_YT | The youtube.com alternative to use when site alternatives are enabled in the config. |
| WHOOGLE_ALT_IG | The instagram.com alternative to use when site alternatives are enabled in the config. |
| WHOOGLE_ALT_RD | The reddit.com alternative to use when site alternatives are enabled in the config. |
### Config Environment Variables
These environment variables allow setting default config values, but can be overwritten manually by using the home page config menu. These allow a shortcut for destroying/rebuilding an instance to the same config state every time.
| Variable | Description |
| ----------------------- | --------------------------------------------------------------- |
| WHOOGLE_CONFIG_COUNTRY | Filter results by hosting country |
| WHOOGLE_CONFIG_LANGUAGE | Set interface and search result language |
| WHOOGLE_CONFIG_DARK | Enable dark theme |
| WHOOGLE_CONFIG_SAFE | Enable safe searches |
| WHOOGLE_CONFIG_ALTS | Use social media site alternatives (nitter, invidious, etc) |
| WHOOGLE_CONFIG_TOR | Use Tor routing (if available) |
| WHOOGLE_CONFIG_NEW_TAB | Always open results in new tab |
| WHOOGLE_CONFIG_GET_ONLY | Search using GET requests only |
| WHOOGLE_CONFIG_URL | The root url of the instance (`https://<your url>/`) |
| WHOOGLE_CONFIG_STYLE | The custom CSS to use for styling (must be single line) |
## Usage
Same as most search engines, with the exception of filtering by time range.
@ -329,6 +354,16 @@ I'm a huge fan of Searx though and encourage anyone to use that instead if they
A lot of the app currently piggybacks on Google's existing support for fetching results pages with Javascript disabled. To their credit, they've done an excellent job with styling pages, but it seems that the image results page - particularly on mobile - is a little rough. Moving forward, with enough interest, I'd like to transition to fetching the results and parsing them into a unique Whoogle-fied interface that I can style myself.
## Public Instances
*Note: Use public instances at your own discretion. Maintainers of Whoogle do not personally validate the integrity of these instances, and popular public instances are more likely to be rate-limited or blocked.*
- [https://whoogle.sdf.org](https://whoogle.sdf.org)
- [https://whoogle.himiko.cloud](https://whoogle.himiko.cloud)
- [https://whoogle.kavin.rocks](https://whoogle.kavin.rocks) or [http://whoogledq5f5wly5p4i2ohnvjwlihnlg4oajjum2oeddfwqdwupbuhqd.onion](http://whoogledq5f5wly5p4i2ohnvjwlihnlg4oajjum2oeddfwqdwupbuhqd.onion)
- [https://search.garudalinux.org](https://search.garudalinux.org)
- [https://whooglesearch.net/](https://whooglesearch.net/)
## Screenshots
#### Desktop
![Whoogle Desktop](docs/screenshot_desktop.jpg)

56
app.json
View File

@ -47,18 +47,68 @@
},
"WHOOGLE_ALT_TW": {
"description": "The site to use as a replacement for twitter.com when site alternatives are enabled in the config.",
"value": "",
"value": "nitter.net",
"required": false
},
"WHOOGLE_ALT_YT": {
"description": "The site to use as a replacement for youtube.com when site alternatives are enabled in the config.",
"value": "",
"value": "invidious.snopyta.org",
"required": false
},
"WHOOGLE_ALT_IG": {
"description": "The site to use as a replacement for instagram.com when site alternatives are enabled in the config.",
"value": "",
"value": "bibliogram.art/u",
"required": false
},
"WHOOGLE_ALT_RD": {
"description": "The site to use as a replacement for reddit.com when site alternatives are enabled in the config.",
"value": "libredd.it",
"required": false
},
"WHOOGLE_CONFIG_COUNTRY": {
"description": "[CONFIG] The country to use for restricting search results (use values from https://raw.githubusercontent.com/benbusby/whoogle-search/develop/app/static/settings/countries.json)",
"value": "",
"required": false
},
"WHOOGLE_CONFIG_LANGUAGE": {
"description": "[CONFIG] The language to use for search results and interface (use values from https://raw.githubusercontent.com/benbusby/whoogle-search/develop/app/static/settings/languages.json)",
"value": "",
"required": false
},
"WHOOGLE_CONFIG_DARK": {
"description": "[CONFIG] Enable dark mode (set to 1 or leave blank)",
"value": "",
"required": false
},
"WHOOGLE_CONFIG_SAFE": {
"description": "[CONFIG] Use safe mode for searches (set to 1 or leave blank)",
"value": "",
"required": false
},
"WHOOGLE_CONFIG_ALTS": {
"description": "[CONFIG] Use social media alternatives (set to 1 or leave blank)",
"value": "",
"required": false
},
"WHOOGLE_CONFIG_TOR": {
"description": "[CONFIG] Use Tor, if available (set to 1 or leave blank)",
"value": "",
"required": false
},
"WHOOGLE_CONFIG_NEW_TAB": {
"description": "[CONFIG] Always open results in new tab (set to 1 or leave blank)",
"value": "",
"required": false
},
"WHOOGLE_CONFIG_GET_ONLY": {
"description": "[CONFIG] Search using GET requests only (set to 1 or leave blank)",
"value": "",
"required": false
},
"WHOOGLE_CONFIG_STYLE": {
"description": "[CONFIG] Custom CSS styling (paste in CSS or leave blank)",
"value": "",
"required": false
}
}
}

33
app/__init__.py
View File

@ -1,30 +1,37 @@
from app.request import send_tor_signal
from app.utils.session_utils import generate_user_keys
from app.utils.gen_ddg_bangs import gen_bangs_json
from app.utils.session import generate_user_key
from app.utils.bangs import gen_bangs_json
from flask import Flask
from flask_session import Session
import json
import os
from stem import Signal
from dotenv import load_dotenv
app = Flask(__name__, static_folder=os.path.dirname(
os.path.abspath(__file__)) + '/static')
app.user_elements = {}
app.default_key_set = generate_user_keys()
# Load .env file if enabled
if os.getenv("WHOOGLE_DOTENV", ''):
dotenv_path = '../whoogle.env'
load_dotenv(os.path.join(os.path.dirname(os.path.abspath(__file__)),
dotenv_path))
app.default_key = generate_user_key()
app.no_cookie_ips = []
app.config['SECRET_KEY'] = os.urandom(32)
app.config['SESSION_TYPE'] = 'filesystem'
app.config['VERSION_NUMBER'] = '0.3.2'
app.config['VERSION_NUMBER'] = '0.4.0'
app.config['APP_ROOT'] = os.getenv(
'APP_ROOT',
os.path.dirname(os.path.abspath(__file__)))
app.config['LANGUAGES'] = json.load(open(
os.path.join(app.config['APP_ROOT'], 'misc/languages.json')))
app.config['COUNTRIES'] = json.load(open(
os.path.join(app.config['APP_ROOT'], 'misc/countries.json')))
app.config['STATIC_FOLDER'] = os.getenv(
'STATIC_FOLDER',
os.path.join(app.config['APP_ROOT'], 'static'))
app.config['LANGUAGES'] = json.load(open(
os.path.join(app.config['STATIC_FOLDER'], 'settings/languages.json')))
app.config['COUNTRIES'] = json.load(open(
os.path.join(app.config['STATIC_FOLDER'], 'settings/countries.json')))
app.config['CONFIG_PATH'] = os.getenv(
'CONFIG_VOLUME',
os.path.join(app.config['STATIC_FOLDER'], 'config'))
@ -40,6 +47,14 @@ app.config['BANG_PATH'] = os.getenv(
app.config['BANG_FILE'] = os.path.join(
app.config['BANG_PATH'],
'bangs.json')
app.config['CSP'] = 'default-src \'none\';' \
'manifest-src \'self\';' \
'img-src \'self\';' \
'style-src \'self\' \'unsafe-inline\';' \
'script-src \'self\';' \
'media-src \'self\';' \
'connect-src \'self\';' \
'form-action \'self\';'
if not os.path.exists(app.config['CONFIG_PATH']):
os.makedirs(app.config['CONFIG_PATH'])

91
app/filter.py
View File

@ -1,14 +1,16 @@
from app.request import VALID_PARAMS
from app.utils.filter_utils import *
from bs4.element import ResultSet
from app.utils.results import *
from bs4 import BeautifulSoup
from bs4.element import ResultSet, Tag
from cryptography.fernet import Fernet
from flask import render_template
import re
import urllib.parse as urlparse
from urllib.parse import parse_qs
class Filter:
def __init__(self, user_keys: dict, mobile=False, config=None):
def __init__(self, user_key: str, mobile=False, config=None) -> None:
if config is None:
config = {}
@ -18,7 +20,7 @@ class Filter:
self.new_tab = config['new_tab'] if 'new_tab' in config else False
self.alt_redirect = config['alts'] if 'alts' in config else False
self.mobile = mobile
self.user_keys = user_keys
self.user_key = user_key
self.main_divs = ResultSet('')
self._elements = 0
@ -29,7 +31,7 @@ class Filter:
def elements(self):
return self._elements
def reskin(self, page):
def reskin(self, page: str) -> str:
# Aesthetic only re-skinning
if self.dark:
page = page.replace(
@ -39,22 +41,18 @@ class Filter:
return page
def encrypt_path(self, msg, is_element=False):
def encrypt_path(self, path, is_element=False) -> str:
# Encrypts path to avoid plaintext results in logs
if is_element:
# Element paths are encrypted separately from text, to allow key
# regeneration once all items have been served to the user
enc_path = Fernet(
self.user_keys['element_key']
).encrypt(msg.encode()).decode()
enc_path = Fernet(self.user_key).encrypt(path.encode()).decode()
self._elements += 1
return enc_path
return Fernet(
self.user_keys['text_key']
).encrypt(msg.encode()).decode()
return Fernet(self.user_key).encrypt(path.encode()).decode()
def clean(self, soup):
def clean(self, soup) -> BeautifulSoup:
self.main_divs = soup.find('div', {'id': 'main'})
self.remove_ads()
self.fix_question_section()
@ -90,7 +88,12 @@ class Filter:
return soup
def remove_ads(self):
def remove_ads(self) -> None:
"""Removes ads found in the list of search result divs
Returns:
None (The soup object is modified directly)
"""
if not self.main_divs:
return
@ -99,7 +102,16 @@ class Filter:
if has_ad_content(_.text)]
_ = div.decompose() if len(div_ads) else None
def fix_question_section(self):
def fix_question_section(self) -> None:
"""Collapses the "People Also Asked" section into a "details" element
These sections are typically the only sections in the results page that
are structured as <div><h2>Title</h2><div>...</div></div>, so they are
extracted by checking all result divs for h2 children.
Returns:
None (The soup object is modified directly)
"""
if not self.main_divs:
return
@ -126,30 +138,33 @@ class Filter:
for question in questions:
question['style'] = 'padding: 10px; font-style: italic;'
def update_element_src(self, element, mime):
element_src = element['src']
if element_src.startswith('//'):
element_src = 'https:' + element_src
elif element_src.startswith(LOGO_URL):
def update_element_src(self, element: Tag, mime: str) -> None:
"""Encrypts the original src of an element and rewrites the element src
to use the "/element?src=" pass-through.
Returns:
None (The soup element is modified directly)
"""
src = element['src']
if src.startswith('//'):
src = 'https:' + src
if src.startswith(LOGO_URL):
# Re-brand with Whoogle logo
element['src'] = 'static/img/logo.png'
element['style'] = 'height:40px;width:162px'
element.replace_with(BeautifulSoup(render_template('logo.html')))
return
elif element_src.startswith(GOOG_IMG):
elif src.startswith(GOOG_IMG) or GOOG_STATIC in src:
element['src'] = BLANK_B64
return
element['src'] = 'element?url=' + self.encrypt_path(
element_src,
src,
is_element=True) + '&type=' + urlparse.quote(mime)
# FIXME: Non-mobile image results link to website instead of image
# if not self.mobile:
# img.append(
# BeautifulSoup(FULL_RES_IMG.format(element_src),
# 'html.parser'))
def update_styling(self, soup):
def update_styling(self, soup) -> None:
""""""
# Remove unnecessary button(s)
for button in soup.find_all('button'):
button.decompose()
@ -172,7 +187,17 @@ class Filter:
except AttributeError:
pass
def update_link(self, link):
def update_link(self, link: Tag) -> None:
"""Update internal link paths with encrypted path, otherwise remove
unnecessary redirects and/or marketing params from the url
Args:
link: A bs4 Tag element to inspect and update
Returns:
None (the tag is updated directly)
"""
# Replace href with only the intended destination (no "utm" type tags)
href = link['href'].replace('https://www.google.com', '')
if 'advanced_search' in href or 'tbm=shop' in href:
@ -212,7 +237,7 @@ class Filter:
# Add no-js option
if self.nojs:
gen_nojs(link)
append_nojs(link)
else:
link['href'] = href

33
app/models/config.py
View File

@ -1,17 +1,24 @@
from flask import current_app
import os
class Config:
def __init__(self, **kwargs):
self.url = ''
self.lang_search = ''
self.lang_interface = ''
self.ctry = ''
self.safe = False
self.dark = False
self.nojs = False
self.tor = False
self.near = ''
self.alts = False
self.new_tab = False
self.get_only = False
app_config = current_app.config
self.url = os.getenv('WHOOGLE_CONFIG_URL', '')
self.lang_search = os.getenv('WHOOGLE_CONFIG_LANGUAGE', '')
self.lang_interface = os.getenv('WHOOGLE_CONFIG_LANGUAGE', '')
self.style = open(os.path.join(app_config['STATIC_FOLDER'],
'css/variables.css')).read()
self.ctry = os.getenv('WHOOGLE_CONFIG_COUNTRY', '')
self.safe = bool(os.getenv('WHOOGLE_CONFIG_SAFE', False))
self.dark = bool(os.getenv('WHOOGLE_CONFIG_DARK', False))
self.alts = bool(os.getenv('WHOOGLE_CONFIG_ALTS', False))
self.nojs = bool(os.getenv('WHOOGLE_CONFIG_NOJS', False))
self.tor = bool(os.getenv('WHOOGLE_CONFIG_TOR', False))
self.near = os.getenv('WHOOGLE_CONFIG_NEAR', '')
self.new_tab = bool(os.getenv('WHOOGLE_CONFIG_NEW_TAB', False))
self.get_only = bool(os.getenv('WHOOGLE_CONFIG_GET_ONLY', False))
self.safe_keys = [
'lang_search',
'lang_interface',
@ -20,6 +27,8 @@ class Config:
]
for key, value in kwargs.items():
if not value:
continue
setattr(self, key, value)
def __getitem__(self, name):

41
app/request.py
View File

@ -23,16 +23,16 @@ class TorError(Exception):
"""Exception raised for errors in Tor requests.
Attributes:
message -- a message describing the error that occurred
disable -- optionally disables Tor in the user config (note:
message: a message describing the error that occurred
disable: optionally disables Tor in the user config (note:
this should only happen if the connection has been dropped
altogether).
"""
def __init__(self, message, disable=False):
def __init__(self, message, disable=False) -> None:
self.message = message
self.disable = disable
super().__init__(self.message)
super().__init__(message)
def send_tor_signal(signal: Signal) -> bool:
@ -64,7 +64,7 @@ def gen_query(query, args, config, near_city=None) -> str:
# Use :past(hour/day/week/month/year) if available
# example search "new restaurants :past month"
sub_lang = ''
lang = ''
if ':past' in query and 'tbs' not in args:
time_range = str.strip(query.split(':past', 1)[-1])
param_dict['tbs'] = '&tbs=' + ('qdr:' + str.lower(time_range[0]))
@ -79,9 +79,10 @@ def gen_query(query, args, config, near_city=None) -> str:
# Example:
# &tbs=qdr:h,lr:lang_1pl
# -- the lr param needs to be extracted and remove the leading '1'
sub_lang = [_ for _ in result_tbs.split(',') if 'lr:' in _]
sub_lang = sub_lang[0][sub_lang[0].find('lr:') +
3:len(sub_lang[0])] if len(sub_lang) > 0 else ''
result_params = [_ for _ in result_tbs.split(',') if 'lr:' in _]
if len(result_params) > 0:
result_param = result_params[0]
lang = result_param[result_param.find('lr:') + 3:len(result_param)]
# Ensure search query is parsable
query = urlparse.quote(query)
@ -103,11 +104,11 @@ def gen_query(query, args, config, near_city=None) -> str:
if 'source' in args:
param_dict['source'] = '&source=' + args.get('source')
param_dict['lr'] = ('&lr=' + ''.join(
[_ for _ in sub_lang if not _.isdigit()]
)) if sub_lang else ''
[_ for _ in lang if not _.isdigit()]
)) if lang else ''
else:
param_dict['lr'] = (
'&lr=' + config.lang_search
'&lr=' + config.lang_search
) if config.lang_search else ''
# 'nfpr' defines the exclusion of results from an auto-corrected query
@ -116,7 +117,7 @@ def gen_query(query, args, config, near_city=None) -> str:
param_dict['cr'] = ('&cr=' + config.ctry) if config.ctry else ''
param_dict['hl'] = (
'&hl=' + config.lang_interface.replace('lang_', '')
'&hl=' + config.lang_interface.replace('lang_', '')
) if config.lang_interface else ''
param_dict['safe'] = '&safe=' + ('active' if config.safe else 'off')
@ -133,9 +134,9 @@ class Request:
search suggestions, and loading of external content (images, audio, etc).
Attributes:
normal_ua -- the user's current user agent
root_path -- the root path of the whoogle instance
config -- the user's current whoogle configuration
normal_ua: the user's current user agent
root_path: the root path of the whoogle instance
config: the user's current whoogle configuration
"""
def __init__(self, normal_ua, root_path, config: Config):
@ -150,12 +151,12 @@ class Request:
# Set up proxy, if previously configured
if os.environ.get('WHOOGLE_PROXY_LOC'):
auth_str = ''
if os.environ.get('WHOOGLE_PROXY_USER'):
auth_str = os.environ.get('WHOOGLE_PROXY_USER') + \
':' + os.environ.get('WHOOGLE_PROXY_PASS')
if os.environ.get('WHOOGLE_PROXY_USER', ''):
auth_str = os.environ.get('WHOOGLE_PROXY_USER', '') + \
':' + os.environ.get('WHOOGLE_PROXY_PASS', '')
self.proxies = {
'http': os.environ.get('WHOOGLE_PROXY_TYPE') + '://' +
auth_str + '@' + os.environ.get('WHOOGLE_PROXY_LOC'),
'http': os.environ.get('WHOOGLE_PROXY_TYPE', '') + '://' +
auth_str + '@' + os.environ.get('WHOOGLE_PROXY_LOC', ''),
}
self.proxies['https'] = self.proxies['http'].replace('http',
'https')

82
app/routes.py
View File

@ -16,8 +16,9 @@ from requests import exceptions
from app import app
from app.models.config import Config
from app.request import Request, TorError
from app.utils.session_utils import valid_user_session
from app.utils.routing_utils import *
from app.utils.bangs import resolve_bang
from app.utils.session import valid_user_session
from app.utils.search import *
# Load DDG bang json files only on init
bang_json = json.load(open(app.config['BANG_FILE']))
@ -53,18 +54,14 @@ def before_request_func():
# Generate session values for user if unavailable
if not valid_user_session(session):
session['config'] = json.load(open(app.config['DEFAULT_CONFIG'])) \
if os.path.exists(app.config['DEFAULT_CONFIG']) else {
'url': request.url_root}
if os.path.exists(app.config['DEFAULT_CONFIG']) else {}
session['uuid'] = str(uuid.uuid4())
session['fernet_keys'] = generate_user_keys(True)
session['key'] = generate_user_key(True)
# Flag cookies as possibly disabled in order to prevent against
# unnecessary session directory expansion
g.cookies_disabled = True
if session['uuid'] not in app.user_elements:
app.user_elements.update({session['uuid']: 0})
# Handle https upgrade
if needs_https(request.url):
return redirect(
@ -87,14 +84,7 @@ def before_request_func():
@app.after_request
def after_request_func(response):
if app.user_elements[session['uuid']] <= 0 and '/element' in request.url:
# Regenerate element key if all elements have been served to user
session['fernet_keys'][
'element_key'] = '' if not g.cookies_disabled else \
app.default_key_set['element_key']
app.user_elements[session['uuid']] = 0
def after_request_func(resp):
# Check if address consistently has cookies blocked,
# in which case start removing session files after creation.
#
@ -108,7 +98,11 @@ def after_request_func(response):
for key in session_list:
session.pop(key)
return response
resp.headers['Content-Security-Policy'] = app.config['CSP']
if os.environ.get('HTTPS_ONLY', False):
resp.headers['Content-Security-Policy'] += 'upgrade-insecure-requests'
return resp
@app.errorhandler(404)
@ -121,22 +115,26 @@ def unknown_page(e):
@auth_required
def index():
# Reset keys
session['fernet_keys'] = generate_user_keys(g.cookies_disabled)
error_message = session[
'error_message'] if 'error_message' in session else ''
session['error_message'] = ''
session['key'] = generate_user_key(g.cookies_disabled)
# Redirect if an error was raised
if 'error_message' in session and session['error_message']:
error_message = session['error_message']
session['error_message'] = ''
return render_template('error.html', error_message=error_message)
return render_template('index.html',
languages=app.config['LANGUAGES'],
countries=app.config['COUNTRIES'],
logo=render_template(
'logo.html',
config=g.user_config),
config=g.user_config,
error_message=error_message,
tor_available=int(os.environ.get('TOR_AVAILABLE')),
version_number=app.config['VERSION_NUMBER'])
@app.route('/opensearch.xml', methods=['GET'])
@auth_required
def opensearch():
opensearch_url = g.app_location
if opensearch_url.endswith('/'):
@ -188,19 +186,16 @@ def autocomplete():
@app.route('/search', methods=['GET', 'POST'])
@auth_required
def search():
# Reset element counter
app.user_elements[session['uuid']] = 0
# Update user config if specified in search args
g.user_config = g.user_config.from_params(g.request_params)
search_util = RoutingUtils(request, g.user_config, session,
cookies_disabled=g.cookies_disabled)
search_util = Search(request, g.user_config, session,
cookies_disabled=g.cookies_disabled)
query = search_util.new_search_query()
resolved_bangs = search_util.bang_operator(bang_json)
if resolved_bangs != '':
return redirect(resolved_bangs)
bang = resolve_bang(query=query, bangs_dict=bang_json)
if bang != '':
return redirect(bang)
# Redirect to home if invalid/blank search
if not query:
@ -208,7 +203,7 @@ def search():
# Generate response and number of external elements from the page
try:
response, elements = search_util.generate_response()
response = search_util.generate_response()
except TorError as e:
session['error_message'] = e.message + (
"\\n\\nTor config is now disabled!" if e.disable else "")
@ -216,27 +211,27 @@ def search():
'tor']
return redirect(url_for('.index'))
if search_util.feeling_lucky or elements < 0:
if search_util.feeling_lucky:
return redirect(response, code=303)
# Keep count of external elements to fetch before
# the element key can be regenerated
app.user_elements[session['uuid']] = elements
# Return 503 if temporarily blocked by captcha
resp_code = 503 if has_captcha(str(response)) else 200
return render_template(
'display.html',
query=urlparse.unquote(query),
search_type=search_util.search_type,
dark_mode=g.user_config.dark,
config=g.user_config,
response=response,
version_number=app.config['VERSION_NUMBER'],
search_header=(render_template(
'header.html',
dark_mode=g.user_config.dark,
config=g.user_config,
logo=render_template('logo.html'),
query=urlparse.unquote(query),
search_type=search_util.search_type,
mobile=g.user_request.mobile)
if 'isch' not in search_util.search_type else ''))
if 'isch' not in search_util.search_type else '')), resp_code
@app.route('/config', methods=['GET', 'POST', 'PUT'])
@ -287,7 +282,9 @@ def url():
if len(q) > 0 and 'http' in q:
return redirect(q)
else:
return render_template('error.html', query=q)
return render_template(
'error.html',
error_message='Unable to resolve query: ' + q)
@app.route('/imgres')
@ -299,13 +296,12 @@ def imgres():
@app.route('/element')
@auth_required
def element():
cipher_suite = Fernet(session['fernet_keys']['element_key'])
cipher_suite = Fernet(session['key'])
src_url = cipher_suite.decrypt(request.args.get('url').encode()).decode()
src_type = request.args.get('type')
try:
file_data = g.user_request.send(base_url=src_url).content
app.user_elements[session['uuid']] -= 1
tmp_mem = io.BytesIO()
tmp_mem.write(file_data)
tmp_mem.seek(0)
@ -336,7 +332,7 @@ def window():
return render_template('display.html', response=results)
def run_app():
def run_app() -> None:
parser = argparse.ArgumentParser(
description='Whoogle Search console runner')
parser.add_argument(

110
app/static/css/dark-theme.css
View File

@ -1,13 +1,17 @@
html {
background-color: #222 !important;
background: var(--whoogle-dark-page-bg) !important;
}
body {
background-color: #222 !important;
background: var(--whoogle-dark-page-bg) !important;
}
div {
color: #fff !important;
color: var(--whoogle-dark-text) !important;
}
label {
color: var(--whoogle-dark-contrast-text) !important;
}
li a {
@ -15,43 +19,113 @@ li a {
}
li {
color: #fff !important;
color: var(--whoogle-dark-text) !important;
}
textarea {
background: var(--whoogle-dark-page-bg) !important;
color: var(--whoogle-dark-text) !important;
}
a:visited h3 div {
color: #bbbbff !important;
color: var(--whoogle-dark-result-visited) !important;
}
a:link h3 div {
color: #4b8eea !important;
color: var(--whoogle-dark-result-title) !important;
}
a:link div {
color: #aaffaa !important;
color: var(--whoogle-dark-result-url) !important;
}
div span {
color: #bbb !important;
color: var(--whoogle-dark-secondary-text) !important;
}
input {
background-color: #111 !important;
color: #fff !important;
background-color: var(--whoogle-dark-page-bg) !important;
color: var(--whoogle-dark-text) !important;
}
#search-bar {
color: #fff !important;
background-color: #222 !important;
select {
background: var(--whoogle-dark-page-bg) !important;
color: var(--whoogle-dark-text) !important;
}
.search-container {
background-color: #222 !important;
background-color: var(--whoogle-dark-page-bg) !important;
}
.ZINbbc{
background-color: #1a1a1a !important;
.ZINbbc {
background-color: var(--whoogle-dark-result-bg) !important;
}
.bRsWnc{
background-color: #1a1a1a !important;
.bRsWnc {
background-color: var(--whoogle-dark-result-bg) !important;
}
#search-bar {
border: 2px solid var(--whoogle-dark-element-bg) !important;
color: var(--whoogle-dark-text) !important;
}
#search-bar:focus {
color: var(--whoogle-dark-text) !important;
}
#search-submit {
border: 1px solid var(--whoogle-dark-element-bg) !important;
background: var(--whoogle-dark-element-bg) !important;
color: var(--whoogle-dark-contrast-text) !important;
}
.info-text {
color: var(--whoogle-dark-contrast-text) !important;
opacity: 75%;
}
.collapsible {
color: var(--whoogle-dark-element-bg) !important;
}
.collapsible:after {
color: var(--whoogle-dark-element-bg) !important;
}
.active {
background-color: var(--whoogle-dark-element-bg) !important;
color: var(--whoogle-dark-contrast-text) !important;
}
.content {
background-color: var(--whoogle-dark-element-bg) !important;
color: var(--whoogle-contrast-text) !important;
}
.active:after {
color: var(--whoogle-dark-contrast-text);
}
#gh-link {
color: var(--whoogle-dark-element-bg);
}
.autocomplete-items {
border: 1px solid #685e79;
}
.autocomplete-items div {
color: #fff;
background-color: #222;
border-bottom: 1px solid #242424;
}
.autocomplete-items div:hover {
background-color: #404040;
}
.autocomplete-active {
background-color: var(--whoogle-dark-element-bg) !important;
color: var(--whoogle-dark-contrast-text) !important;
}

6
app/static/css/header.css
View File

@ -60,3 +60,9 @@ header {
margin: 15px;
display: block;
}
#main>div:focus-within {
border-radius: 8px;
box-shadow: 0 0 6px 1px #2375e8;
}

130
app/static/css/light-theme.css Normal file
View File

@ -0,0 +1,130 @@
html {
background: var(--whoogle-page-bg) !important;
}
body {
background: var(--whoogle-page-bg) !important;
}
div {
color: var(--whoogle-text) !important;
}
label {
color: var(--whoogle-contrast-text) !important;
}
li a {
color: #4b8eaa !important;
}
li {
color: var(--whoogle-text) !important;
}
textarea {
background: var(--whoogle-page-bg) !important;
color: var(--whoogle-text) !important;
}
select {
background: var(--whoogle-page-bg) !important;
color: var(--whoogle-text) !important;
}
.ZINbbc {
background-color: var(--whoogle-result-bg) !important;
}
.bRsWnc {
background-color: var(--whoogle-result-bg) !important;
}
a:visited h3 div {
color: var(--whoogle-result-visited) !important;
}
a:link h3 div {
color: var(--whoogle-result-title) !important;
}
a:link div {
color: var(--whoogle-result-url) !important;
}
div span {
color: var(--whoogle-secondary-text) !important;
}
input {
background-color: var(--whoogle-page-bg) !important;
color: var(--whoogle-text) !important;
}
#search-bar {
color: var(--whoogle-text) !important;
background-color: var(--whoogle-page-bg);
}
.home-search {
border: 3px solid var(--whoogle-element-bg) !important;
}
.search-container {
background-color: var(--whoogle-page-bg) !important;
}
#search-submit {
border: 1px solid var(--whoogle-element-bg) !important;
background: var(--whoogle-element-bg) !important;
color: var(--whoogle-contrast-text) !important;
}
.info-text {
color: var(--whoogle-contrast-text) !important;
opacity: 75%;
}
.collapsible {
color: var(--whoogle-element-bg) !important;
}
.collapsible:after {
color: var(--whoogle-element-bg) !important;
}
.active {
background-color: var(--whoogle-element-bg) !important;
color: var(--whoogle-contrast-text) !important;
}
.content {
background-color: var(--whoogle-element-bg) !important;
color: var(--whoogle-contrast-text) !important;
}
.active:after {
color: var(--whoogle-contrast-text);
}
#gh-link {
color: var(--whoogle-element-bg);
}
.autocomplete-items {
border: 1px solid #d4d4d4;
}
.autocomplete-items div {
background-color: #fff;
border-bottom: 1px solid #d4d4d4;
}
.autocomplete-items div:hover {
background-color: #e9e9e9;
}
.autocomplete-active {
background-color: var(--whoogle-element-bg) !important;
color: var(--whoogle-contrast-text) !important;
}

17
app/static/css/logo.css Normal file
View File

@ -0,0 +1,17 @@
.cls-1 {
fill: transparent;
}
svg {
height: inherit;
}
a {
height: inherit;
}
@media (max-width: 1000px) {
svg {
margin-top: .7em;
}
}

77
app/static/css/main.css
View File

@ -9,7 +9,12 @@ body {
padding-bottom: 10px;
}
.logo-container {
max-height: 500px;
}
.search-container {
background: transparent !important;
width: 80%;
position: absolute;
top: 50%;
@ -26,29 +31,21 @@ body {
}
#search-bar {
background: transparent !important;
width: 100%;
border: 3px solid #685e79;
padding: 5px;
height: 40px;
outline: none;
font-size: 24px;
color: #685e79;
border-radius: 10px 10px 0 0;
max-width: 600px;
background: rgba(0, 0, 0, 0);
}
#search-bar:focus {
color: #685e79;
}
#search-submit {
width: 100%;
height: 40px;
border: 1px solid #685e79;
background: #685e79 !important;
text-align: center;
color: #fff;
cursor: pointer;
font-size: 20px;
align-content: center;
@ -70,7 +67,6 @@ button::-moz-focus-inner {
.collapsible {
outline: 0;
background-color: rgba(0, 0, 0, 0);
color: #685e79;
cursor: pointer;
padding: 18px;
width: 100%;
@ -81,14 +77,8 @@ button::-moz-focus-inner {
border-radius: 10px 10px 0 0;
}
.active {
background-color: #685e79;
color: white;
}
.collapsible:after {
content: '\002B';
color: #685e79;
font-weight: bold;
float: right;
margin-left: 5px;
@ -96,7 +86,6 @@ button::-moz-focus-inner {
.active:after {
content: "\2212";
color: white;
}
.content {
@ -104,8 +93,6 @@ button::-moz-focus-inner {
max-height: 0;
overflow: hidden;
transition: max-height 0.2s ease-out;
background-color: #685e79;
color: white;
border-radius: 0 0 10px 10px;
}
@ -113,12 +100,6 @@ button::-moz-focus-inner {
padding-bottom: 20px;
}
.ua-span {
color: white;
-webkit-box-decoration-break: clone;
box-decoration-break: clone;
}
.hidden {
display: none;
}
@ -135,3 +116,49 @@ footer {
font-style: italic;
font-size: 12px;
}
#config-style {
resize: none;
overflow-y: scroll;
width: 100%;
height: 100px;
}
.whoogle-logo {
display: none;
}
.whoogle-svg {
width: 80%;
display: block;
margin: auto;
padding-bottom: 10px;
}
.autocomplete {
position: relative;
display: inline-block;
width: 100%;
}
.autocomplete-items {
position: absolute;
border-bottom: none;
border-top: none;
z-index: 99;
/*position the autocomplete items to be the same width as the container:*/
top: 100%;
left: 0;
right: 0;
}
.autocomplete-items div {
padding: 10px;
cursor: pointer;
}
details summary {
padding: 10px;
font-weight: bold;
}

40
app/static/css/search-dark.css
View File

@ -1,40 +0,0 @@
.autocomplete {
position: relative;
display: inline-block;
width: 100%;
}
.autocomplete-items {
position: absolute;
border: 1px solid #685e79;
border-bottom: none;
border-top: none;
z-index: 99;
/*position the autocomplete items to be the same width as the container:*/
top: 100%;
left: 0;
right: 0;
}
.autocomplete-items div {
padding: 10px;
cursor: pointer;
color: #fff;
background-color: #222;
border-bottom: 1px solid #242424;
}
.autocomplete-items div:hover {
background-color: #404040;
}
.autocomplete-active {
background-color: #685e79 !important;
color: #ffffff;
}
details summary {
padding: 10px;
font-weight: bold;
}

12
app/static/css/search.css
View File

@ -6,7 +6,6 @@
.autocomplete-items {
position: absolute;
border: 1px solid #d4d4d4;
border-bottom: none;
border-top: none;
z-index: 99;
@ -20,17 +19,6 @@
.autocomplete-items div {
padding: 10px;
cursor: pointer;
background-color: #fff;
border-bottom: 1px solid #d4d4d4;
}
.autocomplete-items div:hover {
background-color: #e9e9e9;
}
.autocomplete-active {
background-color: #685e79 !important;
color: #ffffff;
}
details summary {

26
app/static/css/variables.css Normal file
View File

@ -0,0 +1,26 @@
/* Colors */
:root {
/* LIGHT THEME COLORS */
--whoogle-logo: #685e79;
--whoogle-page-bg: #ffffff;
--whoogle-element-bg: #685e79;
--whoogle-text: #000000;
--whoogle-contrast-text: #ffffff;
--whoogle-secondary-text: #70757a;
--whoogle-result-bg: #ffffff;
--whoogle-result-title: #1967d2;
--whoogle-result-url: #0d652d;
--whoogle-result-visited: #4b11a8;
/* DARK THEME COLORS */
--whoogle-dark-logo: #685e79;
--whoogle-dark-page-bg: #222222;
--whoogle-dark-element-bg: #685e79;
--whoogle-dark-text: #ffffff;
--whoogle-dark-contrast-text: #000000;
--whoogle-dark-secondary-text: #bbbbbb;
--whoogle-dark-result-bg: #000000;
--whoogle-dark-result-title: #1967d2;
--whoogle-dark-result-url: #4b11a8;
--whoogle-dark-result-visited: #bbbbff;
}

19
app/static/img/favicon/manifest.json
View File

@ -1,41 +1,44 @@
{
"name": "App",
"name": "Whoogle Search",
"short_name": "Whoogle",
"display": "fullscreen",
"scope": "/",
"icons": [
{
"src": "\/android-icon-36x36.png",
"src": "android-icon-36x36.png",
"sizes": "36x36",
"type": "image\/png",
"density": "0.75"
},
{
"src": "\/android-icon-48x48.png",
"src": "android-icon-48x48.png",
"sizes": "48x48",
"type": "image\/png",
"density": "1.0"
},
{
"src": "\/android-icon-72x72.png",
"src": "android-icon-72x72.png",
"sizes": "72x72",
"type": "image\/png",
"density": "1.5"
},
{
"src": "\/android-icon-96x96.png",
"src": "android-icon-96x96.png",
"sizes": "96x96",
"type": "image\/png",
"density": "2.0"
},
{
"src": "\/android-icon-144x144.png",
"src": "android-icon-144x144.png",
"sizes": "144x144",
"type": "image\/png",
"density": "3.0"
},
{
"src": "\/android-icon-192x192.png",
"src": "android-icon-192x192.png",
"sizes": "192x192",
"type": "image\/png",
"density": "4.0"
}
]
}
}

1
app/static/img/whoogle.svg Normal file
View File

File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 7.0 KiB

43
app/static/js/controller.js
View File

@ -1,14 +1,3 @@
// Whoogle configurations that use boolean values and checkboxes
CONFIG_BOOLS = [
"nojs", "dark", "safe", "alts", "new_tab", "get_only", "tor"
];
// Whoogle configurations that use string values and input fields
CONFIG_STRS = [
"near", "url"
];
const setupSearchLayout = () => {
// Setup search field
const searchBar = document.getElementById("search-bar");
@ -28,33 +17,6 @@ const setupSearchLayout = () => {
});
};
const fillConfigValues = () => {
// Request existing config info
let xhrGET = new XMLHttpRequest();
xhrGET.open("GET", "config");
xhrGET.onload = function() {
if (xhrGET.readyState === 4 && xhrGET.status !== 200) {
alert("Error loading Whoogle config");
return;
}
// Allow for updating/saving config values
let configSettings = JSON.parse(xhrGET.responseText);
CONFIG_STRS.forEach(function(item) {
let configElement = document.getElementById("config-" + item.replace("_", "-"));
configElement.value = configSettings[item] ? configSettings[item] : "";
});
CONFIG_BOOLS.forEach(function(item) {
let configElement = document.getElementById("config-" + item.replace("_", "-"));
configElement.checked = !!configSettings[item];
});
};
xhrGET.send();
};
const setupConfigLayout = () => {
// Setup whoogle config
const collapsible = document.getElementById("config-collapsible");
@ -69,8 +31,6 @@ const setupConfigLayout = () => {
content.classList.toggle("open");
});
fillConfigValues();
};
const loadConfig = event => {
@ -116,6 +76,9 @@ document.addEventListener("DOMContentLoaded", function() {
setupSearchLayout();
setupConfigLayout();
document.getElementById("config-load").addEventListener("click", loadConfig);
document.getElementById("config-save").addEventListener("click", saveConfig);
// Focusing on the search input field requires a delay for elements to finish
// loading (seemingly only on FF)
setTimeout(function() { document.getElementById("search-bar").focus(); }, 250);

11
app/static/js/header.js Normal file
View File

@ -0,0 +1,11 @@
document.addEventListener("DOMContentLoaded", () => {
const searchBar = document.getElementById("search-bar");
searchBar.addEventListener("keyup", function (event) {
if (event.keyCode !== 13) {
handleUserInput(searchBar);
} else {
document.getElementById("search-form").submit();
}
});
});

44
app/static/js/keyboard.js Normal file
View File

@ -0,0 +1,44 @@
(function () {
let searchBar, results;
const keymap = {
ArrowUp: goUp,
ArrowDown: goDown,
k: goUp,
j: goDown,
'/': focusSearch,
};
let activeIdx = -1;
document.addEventListener('DOMContentLoaded', () => {
searchBar = document.querySelector('#search-bar');
results = document.querySelectorAll('#main>div>div>div>a');
});
document.addEventListener('keydown', (e) => {
if (e.target.tagName === 'INPUT') return true;
if (typeof keymap[e.key] === 'function') {
e.preventDefault();
keymap[e.key]();
}
});
function goUp () {
if (activeIdx > 0) focusResult(activeIdx - 1);
else focusSearch();
}
function goDown () {
if (activeIdx < results.length - 1) focusResult(activeIdx + 1);
}
function focusResult (idx) {
activeIdx = idx;
results[activeIdx].scrollIntoView({ behavior: 'smooth', block: 'center', inline: 'nearest' });
results[activeIdx].focus();
}
function focusSearch () {
activeIdx = -1;
searchBar.focus();
}
}());

8
app/static/js/utils.js
View File

@ -28,7 +28,7 @@ const checkForTracking = () => {
/^[0-9]{15}$/
]
}
}
};
// Creates a link to a UPS/USPS/FedEx tracking page
const createTrackingLink = href => {
@ -37,7 +37,7 @@ const checkForTracking = () => {
link.innerHTML = "View Tracking Info";
link.href = href;
mainDiv.prepend(link);
}
};
// Compares the query against a set of regex patterns
// for tracking numbers
@ -48,12 +48,12 @@ const checkForTracking = () => {
return true;
}
});
}
};
for (const key of Object.keys(matchTracking)) {
compareQuery(matchTracking[key]);
}
}
};
document.addEventListener("DOMContentLoaded", function() {
checkForTracking();

0
app/misc/countries.json → app/static/settings/countries.json
View File

0
app/misc/languages.json → app/static/settings/languages.json
View File

15
app/templates/display.html
View File

@ -5,13 +5,11 @@
<link rel="search" href="opensearch.xml" type="application/opensearchdescription+xml" title="Whoogle Search">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta name="referrer" content="no-referrer">
<script type="text/javascript" src="static/js/autocomplete.js"></script>
<script type="text/javascript" src="static/js/utils.js"></script>
<link rel="stylesheet" href="static/css/{{ 'search-dark' if dark_mode else 'search' }}.css">
<link rel="stylesheet" href="static/css/search.css">
<link rel="stylesheet" href="static/css/variables.css">
<link rel="stylesheet" href="static/css/header.css">
{% if dark_mode %}
<link rel="stylesheet" href="static/css/dark-theme.css"/>
{% endif %}
<link rel="stylesheet" href="static/css/{{ 'dark' if config.dark else 'light' }}-theme.css"/>
<style>{{ config.style }}</style>
<title>{{ query }} - Whoogle Search</title>
</head>
<body>
@ -19,9 +17,12 @@
{{ response|safe }}
</body>
<footer>
<p style="color: {{ '#fff' if dark_mode else '#000' }};">
<p style="color: {{ '#fff' if config.dark else '#000' }};">
Whoogle Search v{{ version_number }} ||
<a style="color: #685e79" href="https://github.com/benbusby/whoogle-search">View on GitHub</a>
</p>
</footer>
<script src="static/js/autocomplete.js"></script>
<script src="static/js/utils.js"></script>
<script src="static/js/keyboard.js"></script>
</html>

2
app/templates/error.html
View File

@ -1,6 +1,6 @@
<h1>Error</h1>
<hr>
<p>
Error parsing "{{ query }}"
Error: "{{ error_message|safe }}"
</p>
<a href="/">Return Home</a>

34
app/templates/header.html
View File

@ -4,15 +4,17 @@
<form class="Pg70bf" id="search-form" method="POST">
<a class="logo-link mobile-logo"
href="/"
style="display:flex; justify-content:center; align-items:center; color:#685e79; font-size:18px; ">
<span style="color: #685e79">Whoogle</span>
style="display:flex; justify-content:center; align-items:center;">
<div style="height: 1.75em;">
{{ logo|safe }}
</div>
</a>
<div class="H0PQec" style="width: 100%;">
<div class="sbc esbc autocomplete">
<input id="search-bar" autocapitalize="none" autocomplete="off" class="noHIxc" name="q"
style="background-color: {{ '#000' if dark_mode else '#fff' }};
color: {{ '#685e79' if dark_mode else '#000' }};
border: {{ '1px solid #685e79' if dark_mode else '' }}"
style="background-color: {{ 'var(--whoogle-dark-result-bg)' if config.dark else 'var(--whoogle-result-bg)' }} !important;
color: {{ 'var(--whoogle-dark-text)' if config.dark else 'var(--whoogle-text)' }};
border: {{ '2px solid var(--whoogle-dark-element-bg)' if config.dark else '' }}; border-radius: 8px;"
spellcheck="false" type="text" value="{{ query }}">
<input name="tbm" value="{{ search_type }}" style="display: none">
<input type="submit" style="display: none;">
@ -26,7 +28,9 @@
<header>
<div class="logo-div">
<a class="logo-link" href="/">
<span style="color: #685e79">Whoogle</span>
<div style="height: 1.65em;">
{{ logo|safe }}
</div>
</a>
</div>
<div class="search-div">
@ -35,9 +39,9 @@
<div style="width: 100%; display: flex">
<input id="search-bar" autocapitalize="none" autocomplete="off" class="noHIxc" name="q"
spellcheck="false" type="text" value="{{ query }}"
style="background-color: {{ '#000' if dark_mode else '#fff' }};
color: {{ '#685e79' if dark_mode else '#000' }};
border: {{ '1px solid #685e79' if dark_mode else '' }}">
style="background-color: {{ 'var(--whoogle-dark-result-bg)' if config.dark else 'var(--whoogle-result-bg)' }} !important;
color: {{ 'var(--whoogle-dark-text)' if config.dark else 'var(--whoogle-text)' }};
border: {{ '2px solid var(--whoogle-dark-element-bg)' if config.dark else '' }}; border-radius: 8px;">
<input name="tbm" value="{{ search_type }}" style="display: none">
<input type="submit" style="display: none;">
<div class="sc"></div>
@ -48,14 +52,4 @@
</header>
{% endif %}
<script>
const searchBar = document.getElementById("search-bar");
searchBar.addEventListener("keyup", function (event) {
if (event.keyCode !== 13) {
handleUserInput(searchBar);
} else {
document.getElementById("search-form").submit();
}
});
</script>
<script type="text/javascript" src="static/js/header.js"></script>

77
app/templates/index.html
View File

@ -21,33 +21,28 @@
<script type="text/javascript" src="static/js/controller.js"></script>
<link rel="search" href="opensearch.xml" type="application/opensearchdescription+xml" title="Whoogle Search">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<link rel="stylesheet" href="static/css/{{ 'search-dark' if config.dark else 'search' }}.css">
<link rel="stylesheet" href="static/css/variables.css">
<link rel="stylesheet" href="static/css/main.css">
{% if config.dark %}
<link rel="stylesheet" href="static/css/dark-theme.css"/>
{% endif %}
<link rel="stylesheet" href="static/css/{{ 'dark' if config.dark else 'light' }}-theme.css"/>
<noscript>
<style>
#main { display: inherit !important; }
.content { max-height: 520px; padding: 18px; border-radius: 10px; }
.content { max-height: 720px; padding: 18px; border-radius: 10px; }
.collapsible { display: none; }
</style>
</noscript>
<style>{{ config.style }}</style>
<title>Whoogle Search</title>
</head>
<body id="main" style="display: none; background-color: {{ '#000' if config.dark else '#fff' }}">
<script>
{% if error_message|length > 0 %}
let error = "{{ error_message|safe }}";
alert(error);
{% endif %}
</script>
<div class="search-container">
<img class="logo" src="static/img/logo.png">
<div class="logo-container">
{{ logo|safe }}
</div>
<form id="search-form" action="search" method="{{ 'get' if config.get_only else 'post' }}">
<div class="search-fields">
<div class="autocomplete">
<input type="text" name="q" id="search-bar" autofocus="autofocus" autocomplete="off">
<input type="text" name="q" id="search-bar" class="home-search" autofocus="autofocus" autocapitalize="none" autocomplete="off">
</div>
<input type="submit" id="search-submit" value="Search">
</div>
@ -57,7 +52,7 @@
<div class="content">
<div class="config-fields">
<form id="config-form" action="config" method="post">
<div class="config-div">
<div class="config-div config-div-ctry">
<label for="config-ctry">Filter Results by Country: </label>
<select name="ctry" id="config-ctry">
{% for ctry in countries %}
@ -71,7 +66,7 @@
</select>
<div><span class="info-text"> — Note: If enabled, a website will only appear in the results if it is *hosted* in the selected country.</span></div>
</div>
<div class="config-div">
<div class="config-div config-div-lang">
<label for="config-lang-interface">Interface Language: </label>
<select name="lang_interface" id="config-lang-interface">
{% for lang in languages %}
@ -84,7 +79,7 @@
{% endfor %}
</select>
</div>
<div class="config-div">
<div class="config-div config-div-search-lang">
<label for="config-lang-search">Search Language: </label>
<select name="lang_search" id="config-lang-search">
{% for lang in languages %}
@ -97,48 +92,52 @@
{% endfor %}
</select>
</div>
<div class="config-div">
<div class="config-div config-div-near">
<label for="config-near">Near: </label>
<input type="text" name="near" id="config-near" placeholder="City Name">
<input type="text" name="near" id="config-near" placeholder="City Name" value="{{ config.near }}">
</div>
<div class="config-div">
<div class="config-div config-div-nojs">
<label for="config-nojs">Show NoJS Links: </label>
<input type="checkbox" name="nojs" id="config-nojs">
<input type="checkbox" name="nojs" id="config-nojs" {{ 'checked' if config.nojs else '' }}>
</div>
<div class="config-div">
<div class="config-div config-div-dark">
<label for="config-dark">Dark Mode: </label>
<input type="checkbox" name="dark" id="config-dark">
<input type="checkbox" name="dark" id="config-dark" {{ 'checked' if config.dark else '' }}>
</div>
<div class="config-div">
<div class="config-div config-div-safe">
<label for="config-safe">Safe Search: </label>
<input type="checkbox" name="safe" id="config-safe">
<input type="checkbox" name="safe" id="config-safe" {{ 'checked' if config.safe else '' }}>
</div>
<div class="config-div">
<div class="config-div config-div-alts">
<label class="tooltip" for="config-alts">Replace Social Media Links: </label>
<input type="checkbox" name="alts" id="config-alts">
<div><span class="info-text"> — Replaces Twitter/YouTube/Instagram links
with Nitter/Invidious/Bibliogram links.</span></div>
<input type="checkbox" name="alts" id="config-alts" {{ 'checked' if config.alts else '' }}>
<div><span class="info-text"> — Replaces Twitter/YouTube/Instagram/Reddit links
with Nitter/Invidious/Bibliogram/Libreddit links.</span></div>
</div>
<div class="config-div">
<div class="config-div config-div-new-tab">
<label for="config-new-tab">Open Links in New Tab: </label>
<input type="checkbox" name="new_tab" id="config-new-tab">
<input type="checkbox" name="new_tab" id="config-new-tab" {{ 'checked' if config.new_tab else '' }}>
</div>
<div class="config-div">
<div class="config-div config-div-tor">
<label for="config-tor">Use Tor: {{ '' if tor_available else 'Unavailable' }}</label>
<input type="checkbox" name="tor" id="config-tor" {{ '' if tor_available else 'hidden' }}>
<input type="checkbox" name="tor" id="config-tor" {{ '' if tor_available else 'hidden' }} {{ 'checked' if config.tor else '' }}>
</div>
<div class="config-div">
<div class="config-div config-div-get-only">
<label for="config-get-only">GET Requests Only: </label>
<input type="checkbox" name="get_only" id="config-get-only">
<input type="checkbox" name="get_only" id="config-get-only" {{ 'checked' if config.get_only else '' }}>
</div>
<div class="config-div">
<div class="config-div config-div-root-url">
<label for="config-url">Root URL: </label>
<input type="text" name="url" id="config-url" value="">
<input type="text" name="url" id="config-url" value="{{ config.url }}">
</div>
<div class="config-div config-div-custom-css">
<label for="config-style">Custom CSS:</label>
<textarea name="style" id="config-style" value="">{{ config.style }}</textarea>
</div>
<div class="config-div">
<input type="submit" id="config-load" onclick="loadConfig(event)" value="Load">&nbsp;
<input type="submit" id="config-load" value="Load">&nbsp;
<input type="submit" id="config-submit" value="Apply">&nbsp;
<input type="submit" id="config-submit" onclick="saveConfig(event)" value="Save As...">
<input type="submit" id="config-save" value="Save As...">
</div>
</form>
</div>
@ -147,7 +146,7 @@
<footer>
<p style="color: {{ '#fff' if config.dark else '#000' }};">
Whoogle Search v{{ version_number }} ||
<a style="color: #685e79" href="https://github.com/benbusby/whoogle-search">View on GitHub</a>
<a id="gh-link" href="https://github.com/benbusby/whoogle-search">View on GitHub</a>
</p>
</footer>
</body>

25
app/templates/logo.html Normal file
View File

@ -0,0 +1,25 @@
<link rel="stylesheet" href="static/css/logo.css">
<svg id="Layer_1" class="whoogle-svg" data-name="Layer 1" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1028 254">
<style>
path {
fill: {{ 'var(--whoogle-dark-logo)' if config.dark else 'var(--whoogle-logo)' }};
}
</style>
<defs>
<style>
</style>
</defs>
<path class="cls-1" d="M1197,667H446V413H1474V667H1208a26.41,26.41,0,0,1,4.26-1.16c32.7-3.35,55.65-27.55,56.45-60.44.57-23.65.27-47.33.32-71,0-17.84-.16-35.67.11-53.5.07-4.92-1.57-6.54-6.3-6.11a74.65,74.65,0,0,1-11,0c-3.63-.2-5.18,1.13-5,4.87.22,4.22.05,8.45.05,12.68a6.16,6.16,0,0,1-3.78-2c-20-23.41-53.18-26.6-77.53-7.84-34,26.17-33.8,79.89-7.68,107.44,24.9,26.24,66,24.37,85.69-1.54a14.39,14.39,0,0,1,2.73-2c0,6.94.39,13.22-.08,19.42-1.18,15.5-7.79,28.06-22.32,34.72-15,6.85-30.27,7.21-44-2.92-5.82-4.28-10.1-10.66-15.66-16.71l-19.87,8.29c8.77,16.61,20.28,29.09,38.17,34.48C1187.28,665.12,1192.18,665.92,1197,667ZM447.16,414.27c.39,1.85.57,3,.86,4q25.22,91.07,50.4,182.12c.92,3.32,2.43,4.55,5.92,4.29a82,82,0,0,1,13.48,0c4.6.43,6.56-1.13,8-5.68,12.37-38.63,25-77.15,37.66-115.7.52-1.6,1.26-3.12,1.89-4.67l1.35.06c.81,2.26,1.68,4.51,2.42,6.79q18.62,57.13,37.12,114.31c1.13,3.5,2.61,5.23,6.58,4.89a80.69,80.69,0,0,1,14,0c4.15.37,5.75-1.19,6.79-5.11Q655,518.89,676.57,438.23c2.07-7.78,4.06-15.58,6.24-24-6.92,0-13.07.29-19.19-.11-4.21-.27-5.6,1.31-6.59,5.25q-17.61,70.1-35.6,140.11c-.42,1.61-1.07,3.17-1.62,4.75a10,10,0,0,1-3.16-4.88q-17.11-51.6-34.21-103.21c-1.72-5.19-2.29-12.33-6-14.86-3.9-2.7-10.86-.78-16.45-1.28-4.1-.37-5.73,1.25-7,5.08q-18.7,57.12-37.79,114.11c-.59,1.77-1.43,3.45-2.15,5.18a9.31,9.31,0,0,1-2.68-4.69Q500.5,522.88,490.62,486c-6-22.47-12-45-18.13-67.39-.44-1.63-2-4.13-3.12-4.19C462.13,414.08,454.86,414.27,447.16,414.27ZM1473.38,543.71c-1-8.62-1.16-16.45-2.77-24-5.08-23.65-18.41-40.82-42.31-47.12-24.75-6.52-47.33-2-65,18.14-15.82,18.09-19.77,39.44-16.45,62.6,4,27.73,26.6,52.65,58.1,54.81,21.42,1.46,39.91-3.91,54.24-20.46,3.51-4.05,6.13-8.88,9.54-13.92l-20.94-8.68c-13.71,20.22-30.84,26.7-50.55,19.53-17.08-6.21-29-23.88-27.23-40.92Zm-746-51.07-1.12-.55V414.65H703.69V604.22h23v-6.36c0-21.84-.08-43.68,0-65.52.07-11.59,3.84-21.92,11.82-30.46,9.41-10.07,21.15-11.89,34-8.78,11.13,2.72,17.67,10.23,20.26,21.14a55.72,55.72,0,0,1,1.46,12.34c.13,24,.07,48,.07,72v5.6h23.49v-4.87c0-24.84.05-49.68-.06-74.52a101.29,101.29,0,0,0-1.06-13.91c-2.8-19.45-15.29-34.48-32.34-38.55-21.17-5-39.58-.47-54.11,16.51C729.19,490.07,728.29,491.38,727.34,492.64Zm179.93-22.47c-38.65,0-66.92,28.86-67,68.47-.06,40.49,28.07,70,66.72,70,38.38,0,66.64-29.26,66.67-69C973.71,499.1,946.09,470.21,907.27,470.17Zm82.22,69.31c.57,5.12.76,10.32,1.76,15.35,10.69,53.81,69.71,66.73,104.35,41.39,20.15-14.74,27.8-35.52,27.31-60.14-.88-44.18-40.84-78.15-90-62.12C1006.24,482.67,989.72,508.59,989.49,539.48Zm333.81,64.95V414.62h-22.65V604.43Z" transform="translate(-446 -413)"></path>
<path id="whoogle-g" d="M1197,667c-4.82-1.08-9.72-1.88-14.44-3.3-17.89-5.39-29.4-17.87-38.17-34.48l19.87-8.29c5.56,6.05,9.84,12.43,15.66,16.71,13.75,10.13,29.07,9.77,44,2.92,14.53-6.66,21.14-19.22,22.32-34.72.47-6.2.08-12.48.08-19.42a14.39,14.39,0,0,0-2.73,2c-19.7,25.91-60.79,27.78-85.69,1.54-26.12-27.55-26.3-81.27,7.68-107.44,24.35-18.76,57.56-15.57,77.53,7.84a6.16,6.16,0,0,0,3.78,2c0-4.23.17-8.46-.05-12.68-.19-3.74,1.36-5.07,5-4.87a74.65,74.65,0,0,0,11,0c4.73-.43,6.37,1.19,6.3,6.11-.27,17.83-.08,35.66-.11,53.5,0,23.67.25,47.35-.32,71-.8,32.89-23.75,57.09-56.45,60.44A26.41,26.41,0,0,0,1208,667Zm50-127.58c-.58-4.61-.86-9.29-1.79-13.83a42.26,42.26,0,0,0-37.31-33.75c-16.16-1.75-33.25,8.46-40.62,24.47-5.34,11.62-5.79,23.83-3.48,36.18,5.94,31.62,42.76,45.77,66.74,25.67C1242.58,568.08,1246.76,554.62,1247,539.42Z" transform="translate(-446 -413)"></path>
<path id="whoogle-w" d="M447.16,414.27c7.7,0,15-.19,22.21.19,1.14.06,2.68,2.56,3.12,4.19,6.13,22.44,12.1,44.92,18.13,67.39q9.88,36.84,19.81,73.66a9.31,9.31,0,0,0,2.68,4.69c.72-1.73,1.56-3.41,2.15-5.18q19-57,37.79-114.11c1.25-3.83,2.88-5.45,7-5.08,5.59.5,12.55-1.42,16.45,1.28,3.67,2.53,4.24,9.67,6,14.86q17.14,51.58,34.21,103.21a10,10,0,0,0,3.16,4.88c.55-1.58,1.2-3.14,1.62-4.75q17.87-70,35.6-140.11c1-3.94,2.38-5.52,6.59-5.25,6.12.4,12.27.11,19.19.11-2.18,8.4-4.17,16.2-6.24,24q-21.5,80.68-42.93,161.39c-1,3.92-2.64,5.48-6.79,5.11a80.69,80.69,0,0,0-14,0c-4,.34-5.45-1.39-6.58-4.89q-18.43-57.2-37.12-114.31c-.74-2.28-1.61-4.53-2.42-6.79l-1.35-.06c-.63,1.55-1.37,3.07-1.89,4.67-12.61,38.55-25.29,77.07-37.66,115.7-1.46,4.55-3.42,6.11-8,5.68a82,82,0,0,0-13.48,0c-3.49.26-5-1-5.92-4.29Q473.31,509.34,448,418.3C447.73,417.23,447.55,416.12,447.16,414.27Z" transform="translate(-446 -413)"></path>
<path id="whoogle-e" d="M1473.38,543.71H1370c-1.76,17,10.15,34.71,27.23,40.92,19.71,7.17,36.84.69,50.55-19.53l20.94,8.68c-3.41,5-6,9.87-9.54,13.92-14.33,16.55-32.82,21.92-54.24,20.46-31.5-2.16-54.12-27.08-58.1-54.81-3.32-23.16.63-44.51,16.45-62.6,17.64-20.17,40.22-24.66,65-18.14,23.9,6.3,37.23,23.47,42.31,47.12C1472.22,527.26,1472.43,535.09,1473.38,543.71Zm-26.69-19.8c2.09-14-14.21-30.54-31.43-32.19-22.21-2.13-43.06,13.12-43.63,32.19Z" transform="translate(-446 -413)"></path>
<path id="whoogle-h" d="M727.34,492.64c.95-1.26,1.85-2.57,2.88-3.77,14.53-17,32.94-21.55,54.11-16.51,17,4.07,29.54,19.1,32.34,38.55a101.29,101.29,0,0,1,1.06,13.91c.11,24.84.06,49.68.06,74.52v4.87H794.3v-5.6c0-24,.06-48-.07-72a55.72,55.72,0,0,0-1.46-12.34c-2.59-10.91-9.13-18.42-20.26-21.14-12.81-3.11-24.55-1.29-34,8.78-8,8.54-11.75,18.87-11.82,30.46-.12,21.84,0,43.68,0,65.52v6.36h-23V414.65h22.53v77.44Z" transform="translate(-446 -413)"></path>
<path id="whoogle-o-1" d="M907.27,470.17c38.82,0,66.44,28.93,66.41,69.47,0,39.73-28.29,69-66.67,69-38.65,0-66.78-29.5-66.72-70C840.35,499,868.62,470.13,907.27,470.17Zm43.24,69.26c-.43-3.79-.72-7.61-1.31-11.37-2.94-18.67-19.1-34.56-36.86-36.35-19.93-2-37.94,8.92-45,27.58-3.74,9.85-4.19,20-2.68,30.44,4,27.42,32.55,44.52,57.87,34.41C939.6,577.32,950.2,560.25,950.51,539.43Z" transform="translate(-446 -413)"></path>
<path id="whoogle-o-2" d="M989.49,539.48c.23-30.89,16.75-56.81,43.45-65.52,49.13-16,89.09,17.94,90,62.12.49,24.62-7.16,45.4-27.31,60.14-34.64,25.34-93.66,12.42-104.35-41.39C990.25,549.8,990.06,544.6,989.49,539.48Zm110.22-.09c-.48-4.29-.7-8.62-1.5-12.84-3.43-18.06-19.37-33.16-36.57-34.84-20.05-2-37.75,8.9-45,27.62-3.51,9.06-3.74,18.45-3,28,2.23,27.4,30.07,46.21,55.87,37.67C1088,578.9,1099.32,561.53,1099.71,539.39Z" transform="translate(-446 -413)"></path>
<path id="whoogle-l" d="M1323.3,604.43h-22.65V414.62h22.65Z" transform="translate(-446 -413)"></path>
<path class="cls-1" d="M1247,539.42c-.24,15.2-4.42,28.66-16.46,38.74-24,20.1-60.8,6-66.74-25.67-2.31-12.35-1.86-24.56,3.48-36.18,7.37-16,24.46-26.22,40.62-24.47a42.26,42.26,0,0,1,37.31,33.75C1246.14,530.13,1246.42,534.81,1247,539.42Z" transform="translate(-446 -413)"></path>
<path class="cls-1" d="M1446.69,523.91h-75.06c.57-19.07,21.42-34.32,43.63-32.19C1432.48,493.37,1448.78,509.88,1446.69,523.91Z" transform="translate(-446 -413)"></path>
<path class="cls-1" d="M950.51,539.43c-.31,20.82-10.91,37.89-28,44.71-25.32,10.11-53.89-7-57.87-34.41-1.51-10.43-1.06-20.59,2.68-30.44,7.08-18.66,25.09-29.59,45-27.58,17.76,1.79,33.92,17.68,36.86,36.35C949.79,531.82,950.08,535.64,950.51,539.43Z" transform="translate(-446 -413)"></path>
<path class="cls-1" d="M1099.71,539.39c-.39,22.14-11.74,39.51-30.16,45.6-25.8,8.54-53.64-10.27-55.87-37.67-.78-9.54-.55-18.93,3-28,7.25-18.72,24.95-29.59,45-27.62,17.2,1.68,33.14,16.78,36.57,34.84C1099,530.77,1099.23,535.1,1099.71,539.39Z" transform="translate(-446 -413)"></path>
</svg>
</a>

61
app/utils/bangs.py Normal file
View File

@ -0,0 +1,61 @@
import json
import requests
DDG_BANGS = 'https://duckduckgo.com/bang.v255.js'
def gen_bangs_json(bangs_file: str) -> None:
"""Generates a json file from the DDG bangs list
Args:
bangs_file: The str path to the new DDG bangs json file
Returns:
None
"""
try:
# Request full list from DDG
r = requests.get(DDG_BANGS)
r.raise_for_status()
except requests.exceptions.HTTPError as err:
raise SystemExit(err)
# Convert to json
data = json.loads(r.text)
# Set up a json object (with better formatting) for all available bangs
bangs_data = {}
for row in data:
bang_command = '!' + row['t']
bangs_data[bang_command] = {
'url': row['u'].replace('{{{s}}}', '{}'),
'suggestion': bang_command + ' (' + row['s'] + ')'
}
json.dump(bangs_data, open(bangs_file, 'w'))
def resolve_bang(query: str, bangs_dict: dict) -> str:
"""Transform's a user's query to a bang search, if an operator is found
Args:
query: The search query
bangs_dict: The dict of available bang operators, with corresponding
format string search URLs
(i.e. "!w": "https://en.wikipedia.org...?search={}")
Returns:
str: A formatted redirect for a bang search, or an empty str if there
wasn't a match or didn't contain a bang operator
"""
split_query = query.split(' ')
for operator in bangs_dict.keys():
if operator not in split_query:
continue
return bangs_dict[operator]['url'].format(
query.replace(operator, '').strip())
return ''

26
app/utils/gen_ddg_bangs.py
View File

@ -1,26 +0,0 @@
import json
import requests
def gen_bangs_json(bangs_file):
# Request list
try:
r = requests.get('https://duckduckgo.com/bang.v255.js')
r.raise_for_status()
except requests.exceptions.HTTPError as err:
raise SystemExit(err)
# Convert to json
data = json.loads(r.text)
# Set up a json object (with better formatting) for all available bangs
bangs_data = {}
for row in data:
bang_command = '!' + row['t']
bangs_data[bang_command] = {
'url': row['u'].replace('{{{s}}}', '{}'),
'suggestion': bang_command + ' (' + row['s'] + ')'
}
json.dump(bangs_data, open(bangs_file, 'w'))

92
app/utils/filter_utils.py → app/utils/results.py
View File

@ -3,14 +3,17 @@ import os
import urllib.parse as urlparse
from urllib.parse import parse_qs
SKIP_ARGS = ['ref_src', 'utm']
FULL_RES_IMG = '<br/><a href="{}">Full Image</a>'
SKIP_PREFIX = ['//www.', '//mobile.', '//m.']
GOOG_STATIC = 'www.gstatic.com'
GOOG_IMG = '/images/branding/searchlogo/1x/googlelogo'
LOGO_URL = GOOG_IMG + '_desk'
BLANK_B64 = ('data:image/png;base64,'
'iVBORw0KGgoAAAANSUhEUgAAAAoAAAAKCAQAAAAnOwc2AAAAD0lEQVR42mNkw'
'AIYh7IgAAVVAAuInjI5AAAAAElFTkSuQmCC')
# Ad keywords
BLACKLIST = [
'ad', 'anuncio', 'annuncio', 'annonce', 'Anzeige', '广告', '廣告', 'Reklama',
@ -22,24 +25,54 @@ BLACKLIST = [
SITE_ALTS = {
'twitter.com': os.getenv('WHOOGLE_ALT_TW', 'nitter.net'),
'youtube.com': os.getenv('WHOOGLE_ALT_YT', 'invidious.snopyta.org'),
'instagram.com': os.getenv('WHOOGLE_ALT_IG', 'bibliogram.art/u')
'instagram.com': os.getenv('WHOOGLE_ALT_IG', 'bibliogram.art/u'),
'reddit.com': os.getenv('WHOOGLE_ALT_RD', 'libredd.it')
}
def has_ad_content(element: str):
return element.upper() in (value.upper() for value in BLACKLIST) \
or 'ⓘ' in element
def has_ad_content(element: str) -> bool:
"""Inspects an HTML element for ad related content
Args:
element: The HTML element to inspect
Returns:
bool: True/False for the element containing an ad
"""
return (element.upper() in (value.upper() for value in BLACKLIST)
or 'ⓘ' in element)
def get_first_link(soup):
def get_first_link(soup: BeautifulSoup) -> str:
"""Retrieves the first result link from the query response
Args:
soup: The BeautifulSoup response body
Returns:
str: A str link to the first result
"""
# Replace hrefs with only the intended destination (no "utm" type tags)
for a in soup.find_all('a', href=True):
# Return the first search result URL
if 'url?q=' in a['href']:
return filter_link_args(a['href'])
return ''
def get_site_alt(link: str):
def get_site_alt(link: str) -> str:
"""Returns an alternative to a particular site, if one is configured
Args:
link: A string result URL to check against the SITE_ALTS map
Returns:
str: An updated (or ignored) result link
"""
for site_key in SITE_ALTS.keys():
if site_key not in link:
continue
@ -47,16 +80,28 @@ def get_site_alt(link: str):
link = link.replace(site_key, SITE_ALTS[site_key])
break
return link.replace('www.', '').replace('//m.', '//')
for prefix in SKIP_PREFIX:
link = link.replace(prefix, '//')
return link
def filter_link_args(query_link):
parsed_link = urlparse.urlparse(query_link)
def filter_link_args(link: str) -> str:
"""Filters out unnecessary URL args from a result link
Args:
link: The string result link to check for extraneous URL params
Returns:
str: An updated (or ignored) result link
"""
parsed_link = urlparse.urlparse(link)
link_args = parse_qs(parsed_link.query)
safe_args = {}
if len(link_args) == 0 and len(parsed_link) > 0:
return query_link
return link
for arg in link_args.keys():
if arg in SKIP_ARGS:
@ -65,19 +110,28 @@ def filter_link_args(query_link):
safe_args[arg] = link_args[arg]
# Remove original link query and replace with filtered args
query_link = query_link.replace(parsed_link.query, '')
link = link.replace(parsed_link.query, '')
if len(safe_args) > 0:
query_link = query_link + urlparse.urlencode(safe_args, doseq=True)
link = link + urlparse.urlencode(safe_args, doseq=True)
else:
query_link = query_link.replace('?', '')
link = link.replace('?', '')
return query_link
return link
def gen_nojs(sibling):
def append_nojs(result: BeautifulSoup) -> None:
"""Appends a no-Javascript alternative for a search result
Args:
result: The search result to append a no-JS link to
Returns:
None
"""
nojs_link = BeautifulSoup(features='html.parser').new_tag('a')
nojs_link['href'] = '/window?location=' + sibling['href']
nojs_link['href'] = '/window?location=' + result['href']
nojs_link['style'] = 'display:block;width:100%;'
nojs_link.string = 'NoJS Link: ' + nojs_link['href']
sibling.append(BeautifulSoup('<br><hr><br>', 'html.parser'))
sibling.append(nojs_link)
result.append(BeautifulSoup('<br><hr><br>', 'html.parser'))
result.append(nojs_link)

111
app/utils/routing_utils.py → app/utils/search.py
View File

@ -1,5 +1,5 @@
from app.filter import Filter, get_first_link
from app.utils.session_utils import generate_user_keys
from app.utils.session import generate_user_key
from app.request import gen_query
from bs4 import BeautifulSoup as bsoup
from cryptography.fernet import Fernet, InvalidToken
@ -8,17 +8,51 @@ from typing import Any, Tuple
import os
TOR_BANNER = '<hr><h1 style="text-align: center">You are using Tor</h1><hr>'
CAPTCHA = 'div class="g-recaptcha"'
def needs_https(url: str) -> bool:
https_only = os.getenv('HTTPS_ONLY', False)
"""Checks if the current instance needs to be upgraded to HTTPS
Note that all Heroku instances are available by default over HTTPS, but
do not automatically set up a redirect when visited over HTTP.
Args:
url: The instance url
Returns:
bool: True/False representing the need to upgrade
"""
https_only = bool(os.getenv('HTTPS_ONLY', 0))
is_heroku = url.endswith('.herokuapp.com')
is_http = url.startswith('http://')
return (is_heroku and is_http) or (https_only and is_http)
class RoutingUtils:
def has_captcha(results: str) -> bool:
"""Checks to see if the search results are blocked by a captcha
Args:
results: The search page html as a string
Returns:
bool: True/False indicating if a captcha element was found
"""
return CAPTCHA in results
class Search:
"""Search query preprocessor - used before submitting the query or
redirecting to another site
Attributes:
request: the incoming flask request
config: the current user config settings
session: the flask user session
"""
def __init__(self, request, config, session, cookies_disabled=False):
method = request.method
self.request_params = request.args if method == 'GET' else request.form
@ -31,23 +65,28 @@ class RoutingUtils:
self.search_type = self.request_params.get(
'tbm') if 'tbm' in self.request_params else ''
def __getitem__(self, name):
def __getitem__(self, name) -> Any:
return getattr(self, name)
def __setitem__(self, name, value):
def __setitem__(self, name, value) -> None:
return setattr(self, name, value)
def __delitem__(self, name):
def __delitem__(self, name) -> None:
return delattr(self, name)
def __contains__(self, name):
def __contains__(self, name) -> bool:
return hasattr(self, name)
def new_search_query(self) -> str:
# Generate a new element key each time a new search is performed
self.session['fernet_keys']['element_key'] = generate_user_keys(
cookies_disabled=self.cookies_disabled)['element_key']
"""Parses a plaintext query into a valid string for submission
Also decrypts the query string, if encrypted (in the case of
paginated results).
Returns:
str: A valid query string
"""
q = self.request_params.get('q')
if q is None or len(q) == 0:
@ -55,53 +94,45 @@ class RoutingUtils:
else:
# Attempt to decrypt if this is an internal link
try:
q = Fernet(
self.session['fernet_keys']['text_key']
).decrypt(q.encode()).decode()
q = Fernet(self.session['key']).decrypt(q.encode()).decode()
except InvalidToken:
pass
# Reset text key
self.session['fernet_keys']['text_key'] = generate_user_keys(
cookies_disabled=self.cookies_disabled)['text_key']
# Strip leading '! ' for "feeling lucky" queries
self.feeling_lucky = q.startswith('! ')
self.query = q[2:] if self.feeling_lucky else q
return self.query
def bang_operator(self, bangs_dict: dict) -> str:
for operator in bangs_dict.keys():
if self.query.split(' ')[0] != operator:
continue
def generate_response(self) -> str:
"""Generates a response for the user's query
return bangs_dict[operator]['url'].format(
self.query.replace(operator, '').strip())
return ''
Returns:
str: A string response to the search query, in the form of a URL
or string representation of HTML content.
def generate_response(self) -> Tuple[Any, int]:
"""
mobile = 'Android' in self.user_agent or 'iPhone' in self.user_agent
content_filter = Filter(
self.session['fernet_keys'],
mobile=mobile,
config=self.config)
full_query = gen_query(
self.query,
self.request_params,
self.config,
content_filter.near)
content_filter = Filter(self.session['key'],
mobile=mobile,
config=self.config)
full_query = gen_query(self.query,
self.request_params,
self.config,
content_filter.near)
get_body = g.user_request.send(query=full_query)
# Produce cleanable html soup from response
html_soup = bsoup(content_filter.reskin(get_body.text), 'html.parser')
html_soup.insert(
0,
bsoup(TOR_BANNER, 'html.parser')
if g.user_request.tor_valid else bsoup('', 'html.parser'))
# Indicate whether or not a Tor connection is active
tor_banner = bsoup('', 'html.parser')
if g.user_request.tor_valid:
tor_banner = bsoup(TOR_BANNER, 'html.parser')
html_soup.insert(0, tor_banner)
if self.feeling_lucky:
return get_first_link(html_soup), 1
return get_first_link(html_soup)
else:
formatted_results = content_filter.clean(html_soup)
@ -116,4 +147,4 @@ class RoutingUtils:
continue
link['href'] += param_str
return formatted_results, content_filter.elements
return str(formatted_results)

42
app/utils/session.py Normal file
View File

@ -0,0 +1,42 @@
from cryptography.fernet import Fernet
from flask import current_app as app
REQUIRED_SESSION_VALUES = ['uuid', 'config', 'key']
def generate_user_key(cookies_disabled=False) -> bytes:
"""Generates a key for encrypting searches and element URLs
Args:
cookies_disabled: Flag for whether or not cookies are disabled by the
user. If so, the user can only use the default key
generated on app init for queries.
Returns:
str: A unique Fernet key
"""
if cookies_disabled:
return app.default_key
# Generate/regenerate unique key per user
return Fernet.generate_key()
def valid_user_session(session: dict) -> bool:
"""Validates the current user session
Args:
session: The current Flask user session
Returns:
bool: True/False indicating that all required session values are
available
"""
# Generate secret key for user if unavailable
for value in REQUIRED_SESSION_VALUES:
if value not in session:
return False
return True

24
app/utils/session_utils.py
View File

@ -1,24 +0,0 @@
from cryptography.fernet import Fernet
from flask import current_app as app
REQUIRED_SESSION_VALUES = ['uuid', 'config', 'fernet_keys']
def generate_user_keys(cookies_disabled=False) -> dict:
if cookies_disabled:
return app.default_key_set
# Generate/regenerate unique key per user
return {
'element_key': Fernet.generate_key(),
'text_key': Fernet.generate_key()
}
def valid_user_session(session):
# Generate secret key for user if unavailable
for value in REQUIRED_SESSION_VALUES:
if value not in session:
return False
return True

22
docker-compose.yml
View File

@ -1,9 +1,26 @@
version: "3"
# cant use mem_limit in a 3.x docker-compose file in non swarm mode
# see https://github.com/docker/compose/issues/4513
version: "2.4"
services:
whoogle-search:
image: benbusby/whoogle-search
container_name: whoogle-search
restart: on-failure:5
pids_limit: 50
mem_limit: 256mb
memswap_limit: 256mb
# user debian-tor from tor package
user: '102'
security_opt:
- no-new-privileges
cap_drop:
- ALL
read_only: true
tmpfs:
- /config/:size=10M,uid=102,gid=102,mode=1700
- /var/lib/tor/:size=10M,uid=102,gid=102,mode=1700
- /run/tor/:size=1M,uid=102,gid=102,mode=1700
#environment: # Uncomment to configure environment variables
# Basic auth configuration, uncomment to enable
#- WHOOGLE_USER=<auth username>
@ -19,6 +36,9 @@ services:
#- WHOOGLE_ALT_TW=nitter.net
#- WHOOGLE_ALT_YT=invidious.snopyta.org
#- WHOOGLE_ALT_IG=bibliogram.art/u
#- WHOOGLE_ALT_RD=libredd.it
# Load environment variables from whoogle.env
#- WHOOGLE_DOTENV=1
ports:
- 5000:5000
restart: unless-stopped

29
misc/heroku-regen.sh Executable file
View File

@ -0,0 +1,29 @@
#!/bin/bash
# Assumes this is being executed from a session that has already logged
# into Heroku with "heroku login -i" beforehand.
#
# You can set this up to run every night when you aren't using the
# instance with a cronjob. For example:
# 0 3 * * * /home/pi/whoogle-search/config/heroku-regen.sh <app_name>
HEROKU_CLI_SITE="https://devcenter.heroku.com/articles/heroku-cli"
if ! [[ -x "$(command -v heroku)" ]]; then
echo "Must have heroku cli installed: $HEROKU_CLI_SITE"
exit 1
fi
cd "$(builtin cd "$(dirname "${BASH_SOURCE[0]}")" && pwd -P)/../"
if [[ $# -ne 1 ]]; then
echo -e "Must provide the name of the Whoogle instance to regenerate"
exit 1
fi
APP_NAME="$1"
heroku apps:destroy "$APP_NAME" --confirm "$APP_NAME"
heroku apps:create "$APP_NAME"
heroku container:login
heroku container:push web
heroku container:release web

0
config/tor/start-tor.sh → misc/tor/start-tor.sh
View File

0
config/tor/torrc → misc/tor/torrc
View File

5
requirements.txt
View File

@ -6,12 +6,12 @@ certifi==2020.4.5.1
cffi==1.13.2
chardet==3.0.4
Click==7.0
cryptography==3.2
cryptography==3.3.2
Flask==1.1.1
Flask-Session==0.3.2
idna==2.9
itsdangerous==1.1.0
Jinja2==2.10.3
Jinja2==2.11.3
MarkupSafe==1.1.1
more-itertools==8.3.0
packaging==20.4
@ -31,3 +31,4 @@ urllib3==1.25.9
waitress==1.4.3
wcwidth==0.1.9
Werkzeug==0.16.0
python-dotenv==0.16.0

6
run
View File

@ -12,12 +12,14 @@ SUBDIR="${1:-app}"
export APP_ROOT="$SCRIPT_DIR/$SUBDIR"
export STATIC_FOLDER="$APP_ROOT/static"
mkdir -p "$STATIC_FOLDER"
# Check for regular vs test run
if [[ "$SUBDIR" == "test" ]]; then
# Set up static files for testing
rm -rf "$STATIC_FOLDER"
ln -s "$SCRIPT_DIR/app/static" "$STATIC_FOLDER"
pytest -sv
else
mkdir -p "$STATIC_FOLDER"
python3 -um app \
--host "${ADDRESS:-0.0.0.0}" \
--port "${PORT:-"${EXPOSE_PORT:-5000}"}"

2
setup.py
View File

@ -8,7 +8,7 @@ setuptools.setup(
author='Ben Busby',
author_email='benbusby@protonmail.com',
name='whoogle-search',
version='0.3.2',
version='0.4.0',
include_package_data=True,
install_requires=requirements,
description='Self-hosted, ad-free, privacy-respecting metasearch engine',

4
test/conftest.py
View File

@ -1,5 +1,5 @@
from app import app
from app.utils.session_utils import generate_user_keys
from app.utils.session import generate_user_key
import pytest
import random
@ -18,6 +18,6 @@ def client():
with app.test_client() as client:
with client.session_transaction() as session:
session['uuid'] = 'test'
session['fernet_keys'] = generate_user_keys()
session['key'] = generate_user_key()
session['config'] = {}
yield client

1
test/misc
View File

@ -1 +0,0 @@
../app/misc/

22
test/test_misc.py
View File

@ -1,20 +1,26 @@
from app.utils.session_utils import generate_user_keys, valid_user_session
from cryptography.fernet import Fernet
from app.utils.session import generate_user_key, valid_user_session
def test_generate_user_keys():
keys = generate_user_keys()
assert 'text_key' in keys
assert 'element_key' in keys
assert keys['text_key'] not in keys['element_key']
key = generate_user_key()
assert Fernet(key)
assert generate_user_key() != key
def test_valid_session(client):
assert not valid_user_session({'fernet_keys': '', 'config': {}})
assert not valid_user_session({'key': '', 'config': {}})
with client.session_transaction() as session:
assert valid_user_session(session)
def test_request_key_generation(client):
def test_query_decryption(client):
# FIXME: Handle decryption errors in search.py and rewrite test
# This previously was used to test swapping decryption keys between
# queries. While this worked in theory and usually didn't cause problems,
# they were tied to session IDs and those are really unreliable (meaning
# that occasionally page navigation would break).
rv = client.get('/')
cookie = rv.headers['Set-Cookie']
@ -23,11 +29,9 @@ def test_request_key_generation(client):
with client.session_transaction() as session:
assert valid_user_session(session)
text_key = session['fernet_keys']['text_key']
rv = client.get('/search?q=test+2', headers={'Cookie': cookie})
assert rv._status_code == 200
with client.session_transaction() as session:
assert valid_user_session(session)
assert text_key not in session['fernet_keys']['text_key']

6
test/test_results.py
View File

@ -1,13 +1,13 @@
from bs4 import BeautifulSoup
from app.filter import Filter
from app.utils.session_utils import generate_user_keys
from app.utils.session import generate_user_key
from datetime import datetime
from dateutil.parser import *
def get_search_results(data):
secret_key = generate_user_keys()
soup = Filter(user_keys=secret_key).clean(
secret_key = generate_user_key()
soup = Filter(user_key=secret_key).clean(
BeautifulSoup(data, 'html.parser'))
main_divs = soup.find('div', {'id': 'main'})

9
test/test_routes.py
View File

@ -19,14 +19,21 @@ def test_feeling_lucky(client):
def test_ddg_bang(client):
# Bang at beginning of query
rv = client.get('/search?q=!gh%20whoogle')
assert rv._status_code == 302
assert rv.headers.get('Location').startswith('https://github.com')
rv = client.get('/search?q=!w%20github')
# Move bang to end of query
rv = client.get('/search?q=github%20!w')
assert rv._status_code == 302
assert rv.headers.get('Location').startswith('https://en.wikipedia.org')
# Move bang to middle of query
rv = client.get('/search?q=big%20!r%20chungus')
assert rv._status_code == 302
assert rv.headers.get('Location').startswith('https://www.reddit.com')
def test_config(client):
rv = client.post('/config', data=demo_config)

25
whoogle.env Normal file
View File

@ -0,0 +1,25 @@
# You can set Whoogle environment variables here, but must set
# WHOOGLE_DOTENV=1 in your deployment to enable these values
#WHOOGLE_ALT_TW=nitter.net
#WHOOGLE_ALT_YT=invidious.snopyta.org
#WHOOGLE_ALT_IG=bibliogram.art/u
#WHOOGLE_ALT_RD=libredd.it
#WHOOGLE_USER=""
#WHOOGLE_PASS=""
#WHOOGLE_PROXY_USER=""
#WHOOGLE_PROXY_PASS=""
#WHOOGLE_PROXY_TYPE=""
#WHOOGLE_PROXY_LOC=""
#HTTPS_ONLY=1
#WHOOGLE_CONFIG_COUNTRY=countryUK # See app/static/settings/countries.json for values
#WHOOGLE_CONFIG_LANGUAGE=lang_en # See app/static/settings/languages.json for values
#WHOOGLE_CONFIG_DARK=1 # Dark mode
#WHOOGLE_CONFIG_SAFE=1 # Safe searches
#WHOOGLE_CONFIG_ALTS=1 # Use social media site alternatives
#WHOOGLE_CONFIG_TOR=1 # Use Tor if available
#WHOOGLE_CONFIG_NEW_TAB=1 # Open results in new tab
#WHOOGLE_CONFIG_GET_ONLY=1 # Search using GET requests only
#WHOOGLE_CONFIG_URL=https://<whoogle url>/
#WHOOGLE_CONFIG_STYLE=":root { /* LIGHT THEME COLORS */ --whoogle-background: #d8dee9; --whoogle-accent: #2e3440; --whoogle-text: #3B4252; --whoogle-contrast-text: #eceff4; --whoogle-secondary-text: #70757a; --whoogle-result-bg: #fff; --whoogle-result-title: #4c566a; --whoogle-result-url: #81a1c1; --whoogle-result-visited: #a3be8c; /* DARK THEME COLORS */ --whoogle-dark-background: #222; --whoogle-dark-accent: #685e79; --whoogle-dark-text: #fff; --whoogle-dark-contrast-text: #000; --whoogle-dark-secondary-text: #bbb; --whoogle-dark-result-bg: #000; --whoogle-dark-result-title: #1967d2; --whoogle-dark-result-url: #4b11a8; --whoogle-dark-result-visited: #bbbbff; }"