Local Instance of READTHEDOCS

I've been struggling in setting up a local instance of readthedocs (rtd) with Solr for searching.

After lots of testing I've come up with the following procedure to have a local instance of Read The Docs and a working Solr. 

Bare in mind that this procedure is not for production instances!

I did this procedure on a clean Ubuntu 14.04 desktop installation.

Prerequisites

Make sure that you install the following packages:

• python-virtualenv
• git
• curl
• libxml2-dev
• libxslt1-dev
• zlib1g-dev
• openjdk-7-dev

Most of the steps can be found in the documentation of readthedocs.org. Here I will show you the steps I followed.

Steps

Use three terminals to get this thing working, one for running readthedocs, one for Solr and one for building the files and saving them into Solr.

1. Create a virtualenv and activate it:
   virtualenv venv
   source venv/bin/activate
2. Create a folder in venv and clone the readthedocs repository:
   cd venv
   mkdir sources
   

   git clone https://github.com/rtfd/readthedocs.org.git

3. Install rtd:
   cd readthedocs.org
   pip install -r pip_requirements.txt
4. Build the rtd database:
   cd readthedocs
   ./manage.py syncdb
   Create a superuser when requested.
5. Migrate:
   ./manage.py migrate
6. Set Solr as local backend in the rtd settings:
1. Go into the settings directory and update the files __init__.py, base.py, and sqlite.py.
2. Update the parameter HAYSTACK_CONNECTIONS with:
   

   HAYSTACK_CONNECTIONS = {
       'default': {
           'ENGINE': 'haystack.backends.solr_backend.SolrEngine',
           'URL': 'http://127.0.0.1:8983/solr',
       }
   }

7. Open your second terminal and create and activate virtualenv, identical as in step 1. 
8. Download and extract Solr 3.5.0:
   

   cd venv/sources/
   curl -O http://archive.apache.org/dist/lucene/solr/3.5.0/apache-solr-3.5.0.tgz
   tar xvzf apache-solr-3.5.0.tgz && SOLR_PATH=`pwd`/apache-solr-3.5.0/example

9. Install pysolr and pyquery: pip install pysolr && pip install pyquery.
10. Go to the directory venv/sources/readthedocs.org/readthedocs.
11. Build a Solr-schema: ./manage.py build_solr_schema > $SOLR_PATH/solr/conf/schema.xml.
12. Go back to the directory venv/sources/apache-solr-3.5.0/example and start solr:
    java -jar start.jar
13. In your first terminal, go to the directory venv/sources/readthedocs.org/readthedocs and start the rtd instance:
    ./manage runserver

You are now able to log in to RTD with the created superuser. Go to 127.0.0.1:8000 for the RTD instance or to 127.0.0.1:8000/admin for the Django settings.

The created setup needs to be filled with data, you can do this via a third terminal and add some projects.

In this third terminal, create a virtualenv as described above and then add projects as follows:

1. Go to venv/sources/readthedocs.org/readthedocs.
2. Add a project with the following command: ./manage.py update_repos
• pip
• haystack
• tastypie
• kong
• read-the-docs
• ...
3. Add the content to Solr, to be able to search in the documentation:
   ./manage.py build_files.

When you add other projects, either via the above command, or by importing them via the rtd web interface, run this last command to make them available through Solr.

You have now a local instance of RTD with Solr as search engine.

A lot more needs to be done in order to make it ready for production environments. Solr should be working in a container, a webserver is required, automation needs to be set up for new projects to become automatically available, and so on. Hope to get this done in the near future.

Until then, have fun with your local ReadTheDocs!