Tuesday 7 October 2014

Discrete Fourier Transform For Freuency Analysis

Jekyll - Static Website Generator

 

Introduction

In this article we will look at Jekyll static site generator to generate a static website and host the same on github using github pages.

Background

Jekyll is a simple, blog aware, static site generator.

A static site generator is a utility that generates ready-to-publish static HTML pages from a set of files usually in markdown or HTML which are suitable for deployment directory on any web-server .The blog-aware means that it can support and maintain website with content added in series like that of blogs.

Jekyll is the engine behind GitHub Pages, which enables us to use Jekyll to host the project’s page, blog, or website from GitHub’s servers for free

Github Pages and Jekyll Installation

Let us consider the github project www.github.com/pi19404/pyVision.

Github pages generator gives us the facility to create a website for the project using one of the standard defined themes .This is the easiest way to start a project without having to worry about the HTML themes and CSS files etc and just focus on content writing.

To generate a static webpage of the project go to project settings on github.com enter image description here Click on the Automatic page generator options

This will provide you with a markdown editor where you can enter the contents for the main page of the website.

Enter the name ,content and other details and click on “Continue to Layouts Button” to proceed to HTML theme selection enter image description here Select the desired theme and click on “Publish Page” button to complete the process enter image description here This will lead to github servers hosting the created website on “http://pi19404.github.io/pyVision”

now we are ready to modify and edit the website.The sources of the website are also stored in github repository of the project in the gh-pages branch.

Thus all we have to do to access the sources is to checkout the gh-pages branch and start modifying content on local server and then push changes onto remote github repository.The github servers will regenerated the website and latest changes will be reflected on your website.

The same process can be used to maintain blogs,post or any other content of the website.

Every GitHub Page is run through Jekyll when you push content to gh-pages branch within your repository

Installing Jekyll

Though Jekyll installation on local PC is not necessary.It can be installed in order to preview the website and troubleshoot issues or bugs before pushing the site on GitHub Pages.

  • Ruby - Jekyll requires the Ruby version 1.9.3 or higher.
    • To install ruby easiest way is to download ruby install managers like “rvm - The Ruby Version Manager”
    • Detailed installation instruction for rvm can be found at http://rvm.io/rvm/install or install the software from package manager like synaptic
    • To install Ruby : rvm install 1.9.3
    • This will download the ruby 1.9.3 sources and perform compilation and deployment at “/usr/share/ruby-rvm/”
    • In some cases you many encounter compilation issues due to outdated version of OpenSSL,in which case install the openssl from rvm tool : rvm pkg install openssl
    • This will install the openssl packaged within the rvm installation directory
    • While compiling ruby we can given commandline arguments so that it referes the openSSL package from the rvm install directory and not default system path :

      rvm install 1.9.3 --with-openssl-dir=/usr/share/ruby-rvm/usr

  • Bundler - Bundler is a package manager ,This can be installed as easily

    gem install bundler

  • Jekyll installation
    • Clone the sites repository on the local machine
    • Change the branch to gh-pages if you have created a project website
    • Create a file called GemFile in the directory with the following content

            source 'https://rubygems.org'
            gem 'github-pages'
      
    • Run the command : bundle install for installing Jekyll
    • To ensure that local development environment is same as that of github regularily update the local environment

      bundle update github-pages

  • Running Jekyll : bundle exec jekyll serve The website is accessible for preview at : http://localhost:4000

Layouts and Website Template

Front-matter is just a set of metadata, delineated by three dashes which takes for form of valid YAML content.Any file that contains a front matter block will be processed by Jekyll as a special file. Thus .Jekyll requires that Markdown files have front-matter defined at the top of every file.The front matter can be included in top of markdown or HTML files.

Between the dashed lines you can set predefined variables ( title,layout) or set custom used defined variables.

The present article was written using Markdown syntax and frontmatter included on top of the files was

---
layout: post
title: Jekyll A Static Website Generator
---
Introduction
-------------
In this article we will look at **Jekyll** static site generator to generate a static website and host the same on **github**  using github pages.

The layout tag specifies the template to be used for generating posts.The templates can be makrdown containing HTML contents.

Jekyll uses the Liquid template system .The variables defined in front matter and page contents can be accessed accessed using the Liquid tags both within the files as well as any layouts that page of post relies on.

Let us look at the template for post called post.html and how contents are incorporated using Liquid markup language

---
layout: default
---

{{ page.title }}

{{ content }}
Written on {{ page.date | date: "%B %e, %Y" }}
The layout tag specifies the template to be used for generating posts.The templates can be markdown containing HTML contents. Jekyll uses the [Liquid template system](https://github.com/shopify/liquid/wiki/liquid-for-designers).The variables defined in front matter and page contents can be accessed accessed using the Liquid tags both within the files as well as any layouts that page of post relies on. Let us look at the template for post called **post.html** and how contents are incorporated using Liquid markup language

Thus when a post is created using the template,then the title specified in the frontmatter is the post is accessed via variable page.title.The page content is accessed via variable content. Jekyll provides numerous predefined global variables that you can set in the front matter of a page or post.

Information on some of them can be found at Frontmatter Predefined variables

The rendered html output for above file is

enter link description here

we have seen that in post.html file we have included frontmatter “layout”.This enables us to include the contents of the post.html file into another files as its contents using Liquid markup language.

This enables us to maintain layout and content files separately and we can change the site layout whenever required without making any changes to the content files.The frontmatter predefined variable provides a lot of flexibility in how we can define complex layouts and themes for the website.

There is a index.html file in the project repository that is auto generated by github pages. We modify this file so that it can be used as a base template for all the pages on the website.

we create a _layouts directory in the root folder of the repository.This directory contains all the files that can be accessed by defining the layout variable in the frontmatter of the files.If the layout variable is assigned values post then the file post.html in the _layout directory will be accessed.

we create a file default.html.The default.html file contains the html headers,javascript,stylesheets etc as well as contents to be included in header and footer of pages. Again Liquid markup language is used to specify where the content is be be included


  
    
    
    pyVision by pi19404

    
    
    
    

 
  
  
    
........
{{ "{{" }} content }}
.......

If the file is markdown file then all its contents are inserted in place of contents tag.If the file is HTML then the declaration inside section tag of class content is inserted in place of contents tag.All the pages of the website including the main page index.html contains frontmatter are designed so that generated contents are inserted within default layout.

Posting Blog

All the blog posts reside in the _posts directory.The format of filename is year-month-day-title.ext.This will generate the blogs posts in year/month/day directory of static website. The blog posts can be html or markdown .

Let us create a blog post called 2014-10-03-Jekyll.md

---
layout: post
title: Jekyll A Static Website Generator
---

Introduction
-------------
In this article we will look at **Jekyll** static site generator to generate a static website and host the same on **github**  using github pages.
........

Now we need to provide links to access the blog content from the main page of website.This is done using Jekyll variables and adding the below content in the index.html page

---
layout: default
---


.......
  • Blog Posts -{{ "{{" }} site.url }}
  • {{ for post in site.posts %}}
  • {{ post.title }}

  • {{ endfor %}}

now we launch the website on local machine by executing command

bundle exec jekyll serve

enter image description here

now we push the repository onto github

git push origin gh-pages

access the webpage and observe the similar output as on local server

The markdown used is called GitHub Flavored Markdown,It is different from standard markdown language.The following like gives the difference and highlights the features of Github Flavored markdown language

now that we have created the html,say we want to use the html content on other sites like codeproject or blogger.we can access the html at _site/posts/2013/10/03/Jekyll.html.

Thus we can simply create blog article with content in markdown in suitable directory and not worry about layout rendering and other issues.The same approach can be used for generating any page of website.

Code

The pyVision repository can be found at www.github.com/pi19404/pyvision and the website is hosted at pyvision All the files used in the present article can be found in the gh-pages branch of the repository.

The file for the present article can be found at _posts/2014-10-03-Jekyll.md

The files can also be downloaded from  

 

Python Project Documentation Using Sphinx


Introduction

In this article we will look at how to uses Sphinx  documentation builder for documenting python project.

Background

Code document is essential for a project .It increases the usability of code are make it appealing for use to wide range of users.It is a neccessary tool for developer working in opensource  or enterprise software projects.

reStructuredText is an easy-to-read,  plaintext markup syntax and parser system.The primary goal of reStructuredText is to define and implement a markup syntax for use variety of documentation domains.The  advantage is that markup language  is readable and simple, yet powerful enough to provide the ability to convert reStructuredText documents into useful structured data formats .

Sphinx uses reStructuredText as its markup language, and is a tool to translate a set of reStructuredText  source files into various output formats like HTML and LaTeX.

The current standard tool for documenting python documents is by using Sphinx.
Sphinx  supports automatic generation of module or package documentation based on parsing function headers and extracting doc strings.Such doc strings  are typically referred to as API documentation

There are two principal steps in making API documentation. First, write doc strings in all key classes, methods, and functions using the formatting options defined by Sphinx and second is configuring and using Sphinx for document generation.

Python Docstrings

Python documentation strings (or docstrings) provide a convenient way of associating documentation with Python modules, functions, classes, and methods.

It is a good programming practice to Write docstrings for all public modules, functions, classes, and methods.Docstrings for private modules can be skipped ,however including comments or docstrings as developer notes is always helpful.

A lot of references can be found online describing the format of generic python docstring and good programming and documentation practice like http://legacy.python.org/dev/peps/pep-0008/#documentation-strings or http://legacy.python.org/dev/peps/pep-0257/

Sphinx uses numpy docstring format.A standard guide to numpy/scipy docstring format can be found at https://github.com/numpy/numpy/blob/master/doc/HOWTO_DOCUMENT.rst.txt


NumPy, SciPy, and the scikits follow a common convention for docstrings that provides for consistency, while also allowing the toolchain to produce well-formatted reference guides.The docstring standard uses re-structured text (reST) syntax and is rendered using Sphinx

The are numerous docstring,here we mention only the important once which are freuently used in numpy/scipy documentation

Function Docstring

A documentation string (docstring) is a string that describes a module, function, class, or method definition. The docstring is a special attribute of the object that can be accessed by (object.__doc__) and is surrounded by triple double quotes .

For example
def compute(self,input):

""" function computes the output of the hidden layer for input matrix """

Parameters Docstring

Parameter docstring contains the description of the function arguments, keywords and their respective types.For example
Parameters

----------

input : numpy array,shape=(N,n_in)

:math:`h_{i-1}(x)` is the `input`


The input variable and associated type and desciption are mentioned.

Returns Docstring

Contains the explanation of the returned values and their types.Name of return variable is optional
Returns

-----------

output : numpy array,shape=(N,n_out)

:math:`f(b_k + w_k^T h_{i-1}(x))`

Latex Math

numpydoc docstring format which is used by Sphix allows us to use latex math expression

An way to include inline math is shown in examples in Parameter and Returns Docstring sections.It contains a specifier ":math:" followed by the math expression in back-uotes.

To include math equations in seperate line we can use

:math:`f(b_k + w_k^T h_{i-1}(x))`
or
.. math::
    f(b_k + w_k^T h_{i-1}(x))

There are various other docstring like Raises,See Also,Notes,References,Examples,warning.Refer the following link to get more information on the same https://github.com/numpy/numpy/blob/master/doc/HOWTO_DOCUMENT.rst.txt

The sample output generated by sphinx for the function is shown below
 def compute(self,input):
"""function computes the output of the hidden layer for input matrix Parameters --------- input : numpy array,shape=(N,n_in) :math:`h_{i-1}(x)` is the `input` Returns ----------- output : numpy array,shape=(N,n_out) :math:`f(b_k + w_k^T h_{i-1}(x)) """ #performs affine transformation over input vector linout=numpy.dot(self.W,input.T)+np.reshape(self.b,(self.b.shape[0],1)); #applies non linear activation function over computed linear transformation
self.output=self.activation(linout).T; return self.output;

compute(input)[source]
function computes the output of the hidden layer for input matrix
Parameters:
input : numpy array,shape=(N,n_in)
hi1(x) is the input
Returns:
output : numpy array,shape=(N,n_out)
f(bk+wTkhi1(x))

Installation and Configuration

Since Sphinx is written in the Python language,Thus atleast python version 2.6 is required.

For Ubuntu OS

Easiest way to Install python,numpy,scipy packages are synaptic package manager
NumpyDocs :easy_install numpydoc
Pillow : pip install Pillow
sudo apt-get install python-matplotlib
apt-get install python-sphinx or easy_install sphinx

To generate documentation run sphinx-apidoc utility

sphinx-apidoc -o docs project_dir -F 

will generate the files required for document generation in the docs directory.The source files are referred from the project_dir path.


cd docs

make html

the make command will generate the html documentation in the _build/html directory.The main page can be accessed by opening the index.html file .

Configuration

To enable numpydoc format docstring as well as latex math euations some additional configurations needs to be performed.

Go to the docs directory and open the config.py file in any document editor

Add or Modify the command to contain the path of the source directory.


sys.path.insert(0, os.path.abspath('project_dir'))

If the above command is not present,the sphix document builder will be unable to find the python file and generate blank documentation.

The next change reuired is to add extensions.Locate the extensions keyword in the document and modify it as below to add extensions for math,numpydocs ,autodoc,doctest etc


extensions = [
              'sphinx.ext.autodoc',
              'sphinx.ext.viewcode',
              'matplotlib.sphinxext.mathmpl',
              'matplotlib.sphinxext.only_directives',
              'matplotlib.sphinxext.plot_directive',
              'sphinx.ext.mathjax',
              'numpydoc',
              'sphinx.ext.autodoc',
              'sphinx.ext.doctest',
              
              
              
]


The file also contains the information like "project name" ,author name ,copyright information which can be modified

project = 'PyVision'
copyright = u'2014, pi19404'

version = '0.0.1'

release = '0.0.1'

# The name of the Pygments (syntax highlighting) style to use.
pygments_style = 'colorful'

There are also a host of options for html formatting

just run make html in the docs directory for changes to reflect and regenerate all the documentation
A html themes directory is taken from scikit-learn source just for demonstration purposes
The following documentation is used for pyVision - python machine learning library project.

Code

The source code for the project can be found at github repository www.github.com/pi19404/pyVision
The doc/api directory contains the code documentation

To generate the documentation for the project go to the api directory and run the command "make html"

you can access the documentation by opening the pyVision/docs/api/_build/html/index.html file in the browser

The html template files have been taken from scikit-learn documentation directory . Documentation of some of the files may be incomplete.

The entire project source can be found at below link

Git Subtree

Introduction

In this article we will look at how to use  Git Subtree Tool to split a sub-directories of a project into separate project

Git Subtree

It often happens that while working on a project,you need to reference another project within it.It may be sources of a third party library or a that particular submodule needs to be treated as seperate project .

Subtrees allow subprojects to be included within a subdirectory of the main project, optionally including the subproject's entire history .

git subtree feature enables user to either Merge subtrees together or split repository into subtrees
Let us consider the repository OpenVision that can be found at www.github.com/pi19404/OpenVision
let us say we want to extract the directory ImgML/pyVision as a separate project.
First step is to clone  the OpenVision repository on local PC

git clone https://github.com/pi19404/openVision.git
Then create the new repository on the local PC
mkdir pyVision

cd pyVision
Initialize a empty git repository and associate it with remote repository on github
git init --bare 

git remote add origin https://github.com/pi19404/pyVision.git
git push -u origin master
now we have two independent repositories.
in the parent repository we run the git subtree split command to create a subtree
git subtree split --prefix ImgML/PyVision --annotate="(pyVision)" -b pyVision
Created branch 'pyVision'
c28961fc63dbeb964955ef50b6d84b63de1d4094

now we create a remote repository
https://github.com/pi19404/pyVision.git

and now we push the subtree changes onto the remote repository using subtree push command
git subtree push --prefix=ImgML/PyVision pyVision master
git push using:  pyVision master
Username for 'https://github.com': pi19404
Password for 'https://pi19404@github.com': 
Counting objects: 43, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (43/43), done.
Writing objects:  25% (11/43), 21.11 MiB | 50.00 KiB/s    
Writing objects: 100% (43/43), 24.78 MiB | 26.00 KiB/s, done.
Total 43 (delta 18), reused 0 (delta 0)
remote: warning: GH001: Large files detected.
remote: warning: See http://git.io/iEPt8g for more information.
remote: warning: File MLP.pyvision1 is 54.42 MB; this is larger than GitHub's recommended maximum file size of 50 MB
To https://github.com/pi19404/pyVision
 * [new branch]      ae84838a2975d3a51efbab7f7c0fbcb94783c056 -> master



now we commit the changes in pyVision repository to remote github.we can see all the files committed in the parent repository along with the github history present in the new repository

pi19404  authored 

now there can be two cases of updating the project in the parent or separated repository 
To avoid any issues,we remove the subdirectory and create is again the parent directory

git rm ImgML/PyVision

git subtree add --prefix=ImgML/PyVision pyVision master

commit 6c22aa8f1c11b201ff03f057963d5b4063c85457
Merge: 7676342 6ce46a5
Author: pi19404 
Date:   Tue Oct 7 01:58:43 2014 +0530

    Add 'ImgML/PyVision/' from commit '6ce46a56b2052fc549f0db3e6e52f56083fb0e64'
    
    git-subtree-dir: ImgML/PyVision
    git-subtree-mainline: 7676342aa84a7b4d8bfffc655d6f0ed9af1a7eda
    git-subtree-split: 6ce46a56b2052fc549f0db3e6e52f56083fb0e64


Changes in parent Directory

The above git log comments are essential when pulling commits from subdiretory remote repository

Changes in Parent Directory

Let us make some changes in the parent repository,remove the files test_parser.py,test_parser3.py,test_plot.pyt,t3.py,sgd.py,parsetab.py,m1.py,calclex.py,bac.py,LoadDatasets.pyb

 git commit -m "removing unwanted files" -a
[master 4156a4d] removing unwanted files

git push origin master

will commit if any changes to master branch,but the subdirectory branch remains unaffected.
To push changes to subdirectory branch we execute


git subtree push --prefix=ImgML/PyVision pyVision master








Changes in the subdirectory

Let us make some changes in the sub directory and we want the changes to reflect in the parent directory

we push multiple commits onto the remote subdirectory and merge them in parent directory and maintaining all the changes in remote as a single commit using the squash directive

  1.  authored 


we pull the changes from the parent directory
git subtree pull --prefix=ImgML/PyVision pyVision master --squash

Installation

you can find instructions on how to install git subtree from sources at link http://engineeredweb.com/blog/how-to-install-git-subtree/ since git subtree changes were introduced in git version 1.7.11 it may not be available in ubuntu repositories by default.
Else you can install latest version of git
sudo add-apt-repository ppa:git-core/ppa 
use synaptic ,update the sources by clicking on reload and install the latest version of git

git subtree sciript can be found at /usr/share/doc/git/contrib/subtree/git-subtree.sh