Build the reference manual incrementally

[qed777]

Building the Sage reference manual can take a significant amount of time. Decreasing this time could speed up Sage development.

The patch is large, but most of it consists of moving files from one location to another, as described below. A summary of the changes:

Changes in doc/en/reference — this is where the size of the patch comes from, although the changes are pretty simple:

• rearrange the directory doc/en/reference: for each file like algebras.rst, create a subdirectory algebras and move algebras.rst to algebras/index.rst. Also create a file algebras/conf.py for the build configuration. All of these new conf.py files are identical. Deal with the contents of the directory reference/media similarly, moving the pictures to the appropriate subdirectory.
• modify reference/index.rst to point to these new files.
• reorganize reference/index.rst so it is arranged, at least somewhat, by topic.
• add intersphinx to conf.py — see below. Also add the new subdirectories to the list exclude_trees.
• new file conf_sub.py, configuration for the pieces of the documentation (as opposed to the main conf.py, which is for building reference/index.rst). This file is imported by each of the files SUBDIRECTORY/conf.py.

Changes to doc/common/builder.py:

• add code to build the reference manual in sections, and also to build the sections in parallel. The reference manual ought to be built twice to resolve references now, so typing "sage -docbuild all html" will build it twice (along with all of the other documents, of course). "sage -docbuild reference html" will just build it once. You can also run "sage -docbuild reference/combinat html", for example, to just build one part of the manual.
• the different parts of the manual are separate documents as far as sphinx is concerned, so allow them to reference each other using the "intersphinx" extension. (This is why we need to build it twice: the first pass assembles the intersphinx databases, the second pass uses the databases to create the references correctly.)
• to accomodate the changes in Add todo extension to Sphinx #11251, which don't seem to be easily compatible with intersphinx, search through the output files looking for "todo" items, and accumulate them in one master "todo" list.
• for pdf format, since it now produces 30 different pdf files, write an html file which links to each of them.

Other changes:

• doc/common/conf.py: add the intersphinx extension to the build process.
• doc/common/themes/sage/layout.html: fix a bug where clicking the Sage logo in the upper left corner of the docs wouldn't take you to the right place, at least in the local documentation.
• doc/common/themes/sageref/: add a new theme for the pieces of the reference manual. This is almost identical to the "sage" theme, except for a little tinkering to the links along the top and bottom lines.
• in the main Sage library, change a few pathnames to media files in the reference manual, since those files have been moved.
• make the necessary changes to .hgignore and MANIFEST.in to deal with the relocated files.

---

The html docs for Sage 5.4.rc2,
html without MathJax,
html with MathJax, and
pdf,
built after applying the patches here.

---

Apply:

• attachment: trac_6495-part1-moving-files-link.patch (or run attachment: trac_6495-script-jhp-link.sh)

• attachment: trac_6495-part2-everything-else.patch

• attachment: trac_6495-part3-the-remaining-vs-5.7.beta4.patch

• attachment: trac_6495-part4-interrupts.patch

• attachment: trac-6495_silence_warning-fh.v2.patch.

• attachment: trac_6495-redirect_html.2.patch.

• attachment: trac_6495-docstrings.patch.

• attachment: trac_6495_separate_inventory.patch

• attachment: trac_6495_fixes.patch

• attachment: trac_6495-filtering.patch

• new Sphinx spkg: http://sage.math.washington.edu/home/palmieri/SPKG/sphinx-1.1.2.p2.spkg

---

Before building the docs, you should delete the documentation output directory: rm -rf SAGE_ROOT/devel/sage/doc/output. To test this, you should run sage --docbuild all html and sage --docbuild all pdf. (Note: just running sage --docbuild reference html will probably produce many warnings. If you run it a second time, the warnings should go away.)

Depends on #13064
Depends on #8327
Depends on #13891

Dependencies: 5.7.beta2 + #13064, #8327, #13891

CC: @jhpalmieri @nexttime @nilesjohnson @hivert @mguaypaq @mwhansen

Component: documentation

Keywords: days38

Author: Mitesh Patel, John Palmieri, Florent Hivert

Reviewer: Volker Braun, Florent Hivert

Merged: sage-5.8.beta0

Issue created by migration from https://trac.sagemath.org/ticket/6495

[qed777]

Experimental.

[qed777]

Author: Mitesh Patel

[qed777]

comment:1

Attachment: trac_6495_ref_pdf_pieces.patch.gz

The attached patch is experimental. Notes:

• sage -docbuild reference pdf fails to build arithgroup.pdf, apparently because of the math macro \ZZ in the title. Unfortunately, I don't know how to fix this.
• Since it replaces the top level PDF file with several smaller files, it breaks the patch at add link to PDF manuals in doc/html/index.html #4460.
• It's not clear what happens to cross-ReST document links. I'll try to investigate.

[qed777]

comment:2

On cross-PDF document links: It seems that Sphinx does not produce these. This may OK, since file:// URLs can break easily.

[qed777]

comment:3

On the \ZZ in arithgroup.tex: It seems the problem stems from \@title in

    \ifsphinxpdfoutput
      \begingroup
      % This \def is required to deal with multi-line authors; it               
      % changes \\ to ', ' (comma-space), making it pass muster for             
      % generating document info in the PDF file.                               
      \def\\{, }
      \pdfinfo{
        /Author (\@author)
        /Title (\@title)
      }
      \endgroup
    \fi

in Sphinx's manual.cls. For some reason, the \math* font commands do not work in this context. I gather that \mathbf is preferred, but one workaround is to use

Arithmetic Subgroups of `{\rm SL}_2({\bf Z})`

in place of

Arithmetic Subgroups of `{\rm SL}_2(\ZZ)`

in arithgroup.rst.

[qed777]

Attachment: trac_6495-reference_breakup.patch.gz

Another approach. Depends on #7549. Still experimental. This patch only. sage repo.

[qed777]

comment:4

The new patch may make it possible to build and update reference manual chapters semi-independently. I think we can use the intersphinx extension to fix the cross-chapter references. But we'll need to build the manual twice, a la LaTeX.

To build just a chapter, try, e.g.,

sage -docbuild reference/algebras html -juiv3

Still to do:

• Make a combined index page and search page.
• Check that PDF generation works.
• Combine chapter PDF files into one large [optional] PDF file (with pdfjam's pdfjoin)?
• Use a specific LaTeX doc title in each conf.py.
• Fix the "Arithmetic Subgroups" heading on the top-level page.
• Use a visual, 2D layout for the top-level page? Group by general area? Add icons?
• Get a reply from sphinx-dev about making relative paths work.
• Build docs in parallel (cf. update doc system to jsmath and improve build system (parallel doc builds) #6255) with multiprocessing?
• Replace the "website" PDF link?
• User-friendliness improvements.
• Encourage more compact chapters? It seems that "Combinatorics" takes the most time and memory.
• ...