Local Instance of READTHEDOCS

I've been struggling in setting up a local instance of readthedocs (rtd) with Solr for searching.

After lots of testing I've come up with the following procedure to have a local instance of Read The Docs and a working Solr. 

Bare in mind that this procedure is not for production instances!

I did this procedure on a clean Ubuntu 14.04 desktop installation.

Prerequisites

Make sure that you install the following packages:

• python-virtualenv
• git
• curl
• libxml2-dev
• libxslt1-dev
• zlib1g-dev
• openjdk-7-dev

Most of the steps can be found in the documentation of readthedocs.org. Here I will show you the steps I followed.

Steps

Use three terminals to get this thing working, one for running readthedocs, one for Solr and one for building the files and saving them into Solr.

1. Create a virtualenv and activate it:
   virtualenv venv
   source venv/bin/activate
2. Create a folder in venv and clone the readthedocs repository:
   cd venv
   mkdir sources
   

   git clone https://github.com/rtfd/readthedocs.org.git

3. Install rtd:
   cd readthedocs.org
   pip install -r pip_requirements.txt
4. Build the rtd database:
   cd readthedocs
   ./manage.py syncdb
   Create a superuser when requested.
5. Migrate:
   ./manage.py migrate
6. Set Solr as local backend in the rtd settings:
1. Go into the settings directory and update the files __init__.py, base.py, and sqlite.py.
2. Update the parameter HAYSTACK_CONNECTIONS with:
   

   HAYSTACK_CONNECTIONS = {
       'default': {
           'ENGINE': 'haystack.backends.solr_backend.SolrEngine',
           'URL': 'http://127.0.0.1:8983/solr',
       }
   }

7. Open your second terminal and create and activate virtualenv, identical as in step 1. 
8. Download and extract Solr 3.5.0:
   

   cd venv/sources/
   curl -O http://archive.apache.org/dist/lucene/solr/3.5.0/apache-solr-3.5.0.tgz
   tar xvzf apache-solr-3.5.0.tgz && SOLR_PATH=`pwd`/apache-solr-3.5.0/example

9. Install pysolr and pyquery: pip install pysolr && pip install pyquery.
10. Go to the directory venv/sources/readthedocs.org/readthedocs.
11. Build a Solr-schema: ./manage.py build_solr_schema > $SOLR_PATH/solr/conf/schema.xml.
12. Go back to the directory venv/sources/apache-solr-3.5.0/example and start solr:
    java -jar start.jar
13. In your first terminal, go to the directory venv/sources/readthedocs.org/readthedocs and start the rtd instance:
    ./manage runserver

You are now able to log in to RTD with the created superuser. Go to 127.0.0.1:8000 for the RTD instance or to 127.0.0.1:8000/admin for the Django settings.

The created setup needs to be filled with data, you can do this via a third terminal and add some projects.

In this third terminal, create a virtualenv as described above and then add projects as follows:

1. Go to venv/sources/readthedocs.org/readthedocs.
2. Add a project with the following command: ./manage.py update_repos
• pip
• haystack
• tastypie
• kong
• read-the-docs
• ...
3. Add the content to Solr, to be able to search in the documentation:
   ./manage.py build_files.

When you add other projects, either via the above command, or by importing them via the rtd web interface, run this last command to make them available through Solr.

You have now a local instance of RTD with Solr as search engine.

A lot more needs to be done in order to make it ready for production environments. Solr should be working in a container, a webserver is required, automation needs to be set up for new projects to become automatically available, and so on. Hope to get this done in the near future.

Until then, have fun with your local ReadTheDocs!

Sphinx, ReadTheDocs and Dashes

For my documentation, I often have to document *nix command-line stuff. No problem until you have to document options that come along with commands.

In Sphinx you have the ":command:" directive to show commands in a uniform way. However, if the command has options that are indicated with a double-dash, the double-dash is by default converted to a single dash.

For example: ":command:`./mycommand.py --myoptions options.ini`" would render as "./mycommand.py -myoptions option.ini". Very annoying indeed, since people tend to copy/paste commands and then they are surprised that "it doesn't work".

Weird thing is that in my local Sphinx setup, using 1.3a0, the rendering is correct, but in the default Sphinx version 1.2.x (if I'm not mistaken, also used by ReadTheDocs), I have the single dash. After looking around on the web, found that the parameter "html_use_smartypants" in the Sphinx conf.py file is the culprit. This parameter is by default disabled, but the smartypants is not disabled by Sphinx and ReadTheDocs itself. This smartypants converts the double dash to a single dash or sometimes en-dash (or em-dash). Since the parameter was commented I didn't take notice of it, but I didn't come to the proper rendering.

To actually disable the smartypants, you have to literally set it to False and then I finally have the correct rendering of my double dash.

# If true, SmartyPants will be used to convert quotes and dashes to

# typographically correct entities.

html_use_smartypants = False

/me happy to start the last day of the week.