Blogging with Pelican, Jupyter notebooks and nbconvert

In data science, Jupyter notebooks are a prime source of content. So, why not use notebooks for blog posts? Much of my exploratory work is captured in notebooks, so I gave it a try. You can do this using jupyter nbconvert or by installing a plugin in Pelican.

Pelican uses plugins to generate HTML and CSS to create posts. I chose Pelican-ipynb to process my notebook files. It has more than 300 stars on Github and seemed to fit the needs perfectly. Their Github repo is well documented and provides all the configuration details you need to get started. Pelican-ipynb worked well on my local system, but I started to experience problems once my blog posts were published. Let me take a step back and tell you that plugins for Pelican are installed as git submodules (a repo in repo). Getting the submodules to be installed and used by Github is where I ran into problems. After several attempts to fix the issue, but having no success, I opted for Jupyter's nbconvert. Being new to Pelican, I plan on giving Pelican-ipynb another try once I am more familiar with it. For now, nbconvert converts my notebooks to Markdown by simply running jupyter nbconvert --to=markdown your-notebook.ipynb. Or you could generate HTML, PDF, or a variety of other formats, see jupyter nbconvert --help for options and examples.

Preparing your Post

  1. Pelican uses metadata to tag and title your posts. For markdown documents, you need to include tags to describe each of your posts.

  2. Tags for Markdown or .md files

    Title:
Slug:
Date:
Category:
Tags:
Author:
Summary:

  3. Tags for notebook or .ipynb files. Put these tags into a markdown cell at the top of your notebook.

    ```text
- Title:
- Slug:
- Date:
- Category:
- Tags:
- Author:
- Summary:
```

  4. Checkout content branch on your blog (I use source), add, edit and check-in your new post

Generating and Reviewing Content

  1. Run jupyter nbconvert --to=markdown your-notebook.ipynb. Note: jupyter nbconvert --help has good explanations of how else to use nbconvert
  2. Run pelican content from your blogs root directory and to build the the static files from your content
  3. Fix any warnings or errors that Pelican reports
  4. Change to the output directory by running cd output
  5. Start Python's built-in web server by running: python -m http.server 8000
  6. Open http://localhost:8000 in your browser and review the content and appearance of your blog
  7. Keep editing and refreshing your browser until satisfied
  8. Control-c to halt python -m http.server 8000 process

Publishing to Github Pages

Github pages support both individuals and organizations. For individuals, you must publish your blog using the master branch. So, instead of being the ultimate source of your blog, the master branch is composed of build artifacts generated by Pelican. To me, this counter-intuitive, so this post is intended to provide clarity to all who are interested (my future self included).

  1. Check in any last changes to your source branch
  2. Run pelican content -o output -s pelicanconf.py
  3. Run ghp-import output -b gh-pages
  4. Run git push https://github.com/<username>/<username>.github.com.git gh-pages:master
  5. If you get a ! [rejected] gh-pages -> master (non-fast-forward) message, then run git push -f https://github.com/<username>/<username>.github.com.git gh-pages:master

Resources

  1. Documentation: Pelican Stable
  2. Repository: pelican-ipynb plugin
  3. Blog: Matt Makai's Excellent Pelican Posts
  4. Blog: Some Tips for Using Jupyter Notebooks with Pelican
  5. Blog: Hacking my way to a Jupyter notebook powered blog
  6. Stackoverflow: rejected master -> master (non-fast-forward)