Merge branch 'master' into no-mongo

README.md

@@ -1,12 +1,8 @@
 ## What this is
 
-https://metaforecast.org is a search engine for probabilities from various prediction markes and forecasting platforms (try searching "Trump", "China" or "Semiconductors").
+[Metaforecast](https://metaforecast.org) is a search engine for probabilities from various prediction markes and forecasting platforms. Try searching "Trump", "China" or "Semiconductors".
 
-This repository includes a source code for the website, as well as a set of libraries that fetches probabilities/forecasts.
+This repository includes the source code for both the website and the library that fetches forecasts needed to replace them. We also aim to provide tooling to integrate metaforecast with other services.
-
-We also provide a public database, which can be accessed with a script similar to [this one](./src/backend/manual/manualDownload.js).
-
-I also created a search engine using Elicit's IDE, which uses GPT-3 to deliver vastly superior semantic search (as opposed to fuzzy word matching). If you have access to the Elicit IDE, you can use the action "Search Metaforecast database". However, I'm not currently updating it regularly.
 
 [![](./public/screenshot.png)](https://metaforecast.org)
 
@@ -30,7 +26,7 @@ See [./docs/configuration.md](./docs/configuration.md) for details.
 
 ### 3. Actually run
 
-`npm run cli` starts a local CLI which presents the user with choices; if you would like to skip each step, use the option number instead, e.g., `npm run start 14`.
+`npm run cli` starts a local CLI which presents the user with choices; if you would like to skip each step, use the option number instead, e.g., `npm run cli 14`.
 
 `npm run next-dev` starts a Next.js dev server with the website on `http://localhost:3000`.
 
@@ -43,6 +39,19 @@ $ npm install
 $ node src/backend/manual/manualDownload.js
 ```
 
+## Integrations
+
+Metaforecast has been integrated into:
+
+- Twitter, using our [@metaforecast](https://twitter.com/metaforecast) bot
+- [Global Guessing](https://globalguessing.com/russia-ukraine-forecasts/), which integrates our dashboards
+- [Fletcher](https://fletcher.fun/), a popular Discord bot. You can invoke metaforecast with `!metaforecast search-term`
+- [Elicit](https://elicit.org/), which uses GPT-3 to deliver vastly superior semantic search (as opposed to fuzzy word matching). If you have access to the Elicit IDE, you can use the action "Search Metaforecast database. This is not being updated regularly.
+
+We also provide a public database, which can be accessed with a script similar to [this one](./src/backend/manual/manualDownload.js). We are also open to integrating our Algolia search instance with other trusted services (in addition to Fletcher.)
+
+In general, if you want to integrate metaforecast into your service, we want to hear from you.
+
 ## Code layout
 
 - frontend code is in [src/pages/](./src/pages/), [src/web/](./src/web/) and in a few other places which are required by Next.js (e.g. root-level configs in postcss.config.js and tailwind.config.js)
@@ -62,11 +71,10 @@ Overall, the services which we use are:
 
 - Algolia for search
 - Netlify for website deployment
-- DigitalOcean for background jobs, e.g. fetching new forecasts
+- Heroku for background jobs, e.g. fetching new forecasts
 - Postgres on DigitalOcean for database
 
 ## Various notes
 
 - Commits follow [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/#summary)
-- Right now, I'm fetching only a couple of common properties, such as the title, url, platform, whether a question is binary (yes/no), its percentage, and the number of forecasts.
+- For elicit and metaculus, this library currently filters out questions with <10 predictions.
-- For elicit and metaculus, this library currently filters questions with <10 predictions.

package-lock.json

@@ -1,12 +1,12 @@
 {
   "name": "metaforecast",
-  "version": "1.0.0",
+  "version": "2.0.0",
   "lockfileVersion": 2,
   "requires": true,
   "packages": {
     "": {
       "name": "metaforecast",
-      "version": "1.0.0",
+      "version": "2.0.0",
       "license": "MIT",
       "dependencies": {
         "@tailwindcss/forms": "^0.4.0",
@@ -33,6 +33,7 @@
         "json2csv": "^5.0.5",
         "multiselect-react-dropdown": "^2.0.17",
         "next": "12",
+        "next-plausible": "^3.1.6",
         "nprogress": "^0.2.0",
         "open": "^7.3.1",
         "papaparse": "^5.3.0",
@@ -31952,6 +31953,19 @@
         }
       }
     },
+    "node_modules/next-plausible": {
+      "version": "3.1.6",
+      "resolved": "https://registry.npmjs.org/next-plausible/-/next-plausible-3.1.6.tgz",
+      "integrity": "sha512-9L1l3f59122IBNjVDqZEE2y5kA/a8rk3xYqWWfHH9LLxtWw6hqiL43RQGT7xYjHofGdn+RX3WTVRGIv0Iq/wKA==",
+      "funding": {
+        "url": "https://github.com/4lejandrito/next-plausible?sponsor=1"
+      },
+      "peerDependencies": {
+        "next": "^11.1.0 || ^12.0.0",
+        "react": "^16.8.0 || ^17.0.0",
+        "react-dom": "^16.8.0 || ^17.0.0"
+      }
+    },
     "node_modules/next/node_modules/postcss": {
       "version": "8.4.5",
       "resolved": "https://registry.npmjs.org/postcss/-/postcss-8.4.5.tgz",
@@ -58677,6 +58691,12 @@
         }
       }
     },
+    "next-plausible": {
+      "version": "3.1.6",
+      "resolved": "https://registry.npmjs.org/next-plausible/-/next-plausible-3.1.6.tgz",
+      "integrity": "sha512-9L1l3f59122IBNjVDqZEE2y5kA/a8rk3xYqWWfHH9LLxtWw6hqiL43RQGT7xYjHofGdn+RX3WTVRGIv0Iq/wKA==",
+      "requires": {}
+    },
     "node-abi": {
       "version": "3.8.0",
       "resolved": "https://registry.npmjs.org/node-abi/-/node-abi-3.8.0.tgz",

package.json

@@ -1,6 +1,6 @@
 {
   "name": "metaforecast",
-  "version": "1.0.0",
+  "version": "2.0.0",
   "description": "Get forecasts from various platforms",
   "repository": {
     "type": "git",
@@ -51,6 +51,7 @@
     "json2csv": "^5.0.5",
     "multiselect-react-dropdown": "^2.0.17",
     "next": "12",
+    "next-plausible": "^3.1.6",
     "nprogress": "^0.2.0",
     "open": "^7.3.1",
     "papaparse": "^5.3.0",

src/backend/manual/pullSuperforecastsManually.sh

@@ -1,4 +1,4 @@
 #!/bin/bash
-cd /home/loki/Documents/core/software/fresh/js/metaforecast/metaforecast-backend
+
-date > ./notes/last-superforecast-pull.txt
+cd /home/loki/Documents/core/software/fresh/js/metaforecast/metaforecast-monorepo
-/home/loki/.nvm/versions/node/v16.8.0/bin/node ./src/manual/pullSuperforecastsManually.js >> ./notes/last-superforecast-pull.txt
+/home/loki/.nvm/versions/node/v17.5.0/bin/npm run cli 3 > ../last-superforecast-fetch.txt

src/pages/_app.tsx

@@ -1,8 +1,10 @@
-import 'nprogress/nprogress.css';
+import "nprogress/nprogress.css";
-import '../styles/main.css';
+import "../styles/main.css";
 
-import Router from 'next/router';
+import Router from "next/router";
-import NProgress from 'nprogress';
+import NProgress from "nprogress";
+
+import PlausibleProvider from "next-plausible";
 
 Router.events.on("routeChangeStart", (as, { shallow }) => {
   console.log(shallow);
@@ -14,7 +16,11 @@ Router.events.on("routeChangeComplete", () => NProgress.done());
 Router.events.on("routeChangeError", () => NProgress.done());
 
 function MyApp({ Component, pageProps }) {
-  return <Component {...pageProps} />;
+  return (
+    <PlausibleProvider domain="metaforecast.org">
+      <Component {...pageProps} />
+    </PlausibleProvider>
+  );
 }
 
 export default MyApp;

src/pages/about.tsx

@@ -1,16 +1,16 @@
-import React from 'react';
+import React from "react";
-import ReactMarkdown from 'react-markdown';
+import ReactMarkdown from "react-markdown";
-import gfm from 'remark-gfm';
+import gfm from "remark-gfm";
 
-import Layout from '../web/display/layout';
+import Layout from "../web/display/layout";
 
 let readmeMarkdownText = `# About
 
-This is a search engine for probabilities. Given a query, it searches for relevant questions in various prediction markets and forecasting platforms (namely Betfair, FantasySCOTUS, Foretold, Good Judgment, Good Judgment Open, Guesstimate, GiveWell & OpenPhilanthropy blogposts as of early 2021, Kalshi, Infer, Ladbrokes, Manifold Markets, Metaculus, Polymarket, PredictIt, Smarkets, William Hill, and Michael Aird's database of existential risk estimates). For example, try searching for "China", "North Korea", "Semiconductors", "COVID", "Trump", or "X-risk".
+This webpage is a search engine for probabilities. Given a query, it searches for relevant questions in various prediction markets and forecasting platforms. For example, try searching for "China", "North Korea", "Semiconductors", "COVID", "Trump", or "X-risk". In addition to search, we also provide various [tools](http://localhost:3000/tools).
 
-You can also call the metaforecast database from within Twitter by mentioning @metaforecast, [like so](https://twitter.com/NunoSempere/status/1433160907308294144).
+We are very interested in integrating Metaforecast with other services, and the whole thing is [open source](https://github.com/QURIresearch/metaforecast). So far, Metaforecast has been integrated with [Twitter](https://twitter.com/NunoSempere/status/1433160907308294144), [Fletcher](https://fletcher.fun/), [GlobalGuessing](https://globalguessing.com/russia-ukraine-forecasts/) and previously [Elicit](https://elicit.org/).
 
-You can read a longer writeup with thoughts and motivations [here](https://forum.effectivealtruism.org/posts/tEo5oXeSNcB3sYr8m/introducing-metaforecast-a-forecast-aggregator-and-search), and an update thereto [here](https://www.lesswrong.com/posts/5hugQzRhdGYc6ParJ/metaforecast-update-better-search-capture-functionality-more). You can read the code for the back-end used to fetch probabilities [here](https://github.com/QURIresearch/metaforecast)— including instructions for how to download the underlying dataset, or view the source for the front-end [here](https://github.com/QURIresearch/metaforecast-frontend). 
+You can read a longer writeup with thoughts and motivations [here](https://forum.effectivealtruism.org/posts/tEo5oXeSNcB3sYr8m/introducing-metaforecast-a-forecast-aggregator-and-search), and an update thereto [here](https://www.lesswrong.com/posts/5hugQzRhdGYc6ParJ/metaforecast-update-better-search-capture-functionality-more).
 
 ## Advanced search
 If your initial search doesn't succeed, you might want to try tinkering with the advanced search. In particular, try increasing or decreasing the stars threshold, or changing the number of search results shown.