Development

Useful Resources

IRC channels on chat.freenode.net (quick communication and discussion):

  • #moin-dev (core development topics)

  • #moin (user support, extensions)

Wikis:

Documentation (installation, configuration, user docs, api reference):

Repository, Issue tracker (bugs, proposals, todo), Code Review, etc.:

Requirements for development

The virtualenv Python package is required. The installation process for virtualenv varies with your OS and Python distribution. Many linux distributions have a package manager that may do the installation. Windows users (and perhaps others) may download setuptools from https://pypi.org/project/setuptools/. Once setuptools is installed, do “easy_install virtualenv”. Current ActiveState distributions include virtualenv in the installation bundle. If all else fails, try your favorite search engine.

git is required should you wish to contribute patches to the moin2 development effort. Even if you do not intend to contribute, git is highly recommended as it will make it easy for you to obtain fixes and enhancements from the moin2 repositories. git can be installed with most linux package managers or downloaded from https://git-scm.com/. You can also find GUI clients there.

Typical development workflow

This is the typical workflow for anyone that wants to contribute to the development of Moin2.

create your development environment

  • if you do not have a github account, create one at https://github.com/

  • fork the main repository: https://github.com/moinwiki/moin to your gh user

  • clone your gh repo to your local development machine:

    cd <parent_directory_of_your_future_repo>
git clone https://github.com/yourname/moin.git

  • cd to repo root:

    cd moin

  • create the virtualenv and download packages:

    python quickinstall.py

  • activate virtualenv:

    . activate  # Windows: activate

  • create a wiki instance and load sample data:

    ./m sample  # Windows: m sample

  • start the built-in server:

    ./m run  # Windows: m run

  • point your browser at http://127.0.0.1:8080/ to access your development wiki

  • key ctrl+C to stop the built-in server

add more tools, exercise tools

  • install additional software that developers may require:

    ./m extras  # Windows: m extras

  • run the unit tests, note any existing test failures:

    ./m tests  # Windows: m tests

  • install NodeJS and NPM with Linux package manager; Windows users may download both from https://nodejs.org/download/

    • On Ubuntu 14.04 or any distribution based on Ubuntu you need to install “npm” and “nodejs-legacy” (to get the “node” command).

  • install lessc (“less” below is not a typo):

    sudo npm install less -g  # Windows: npm install less -g
lessc --version"  # show version number to prove it works

  • regenerate CSS files:

    ./m css  # Windows: m css
git diff  # verify nothing changed

  • check for coding errors (tabs, trailing spaces, line endings, template indentation and spacing):

    ./m coding-std  # Windows: m coding-std
git diff  # verify nothing changed

  • revert any changes from above:

    git reset --hard

  • create local docs:

    ./m docs  # Windows: m docs

  • set options on your favorite editor or IDE

    • convert tabs to 4 spaces

    • delete trailing blanks on file save

    • use unix line endings (use Windows line endings on .bat and .cmd files)

    • use mono-spaced font for editing

  • if you are new to git, read about it (https://git-scm.com/book/), consider printing a cheatsheet

  • if you want a Python IDE, try https://www.jetbrains.com/pycharm/ Free Community Edition

  • join #moin-dev IRC channel; ask questions, learn what other developers are doing

review configuration options

  • review https://moin-20.readthedocs.io/en/latest/admin/configure.html

  • following the instructions in wikiconfig.py, create wikiconfig_local.py and wikiconfig_editme.py

  • configure options by editing wikiconfig_editme.py

    • set superuser privileges on at least one username

    • the default configuration options are commonly used, it is likely new bugs can be found by testing different options

find a task to work on

  • look at the issue tracker to find a task you can solve

  • in case you find a new bug or want to work on some (non-trivial) new issue or idea that is not on the issue tracker, create an issue with a detailed description

  • discuss your chosen task with other developers on the #moin-dev IRC channel

  • to avoid duplicate work, add a comment on the issue tracker that you are working on that issue

  • just before you start to code changes, bring your repo up to date:

    git checkout master       # make sure you are on master branch
git pull mm master        # update your master branch
git checkout -b mychange  # create a new branch "mychange"
...                       # implement your change
tox                       # run the tests, fix any new failure!
git status                # check what new files you created
git diff                  # check what changes you did
git add ...               # add the files you want to commit
git commit                # commit, write a nice commit comment
git push                  # push to your gh user's moin repo
...                       # go to gh moinwiki/moin and make a PR

develop a testing strategy

  • if you fix something that had no test, first try to write a correct, but failing test for it, then fix the code and see a successful test

  • if you implement new functionality, write tests for it first, then implement it

  • make a plan for using a browser to test your changes; which wiki pages are effected, how many browsers must be tested

develop a working solution

  • work in your local repo on your local development machine (be sure you work in the right branch)

  • concentrate on one issue / one topic, create a clean set of changes (that means not doing more than needed to fix the issue, but also it means fixing the issue completely and everywhere)

  • write good, clean, easy-to-understand code

  • obey PEP-8

  • do not fix or change code unrelated to your task, if you find unrelated bugs, create new issues on the tracker

  • regularly run the unit tests (“./m tests”), the amount of failing tests shall not increase due to your changes

review your working solution

  • use git diff, git status - read everything you changed - slowly, look for things that can be improved

    • if you have TortoiseGIT, use those graphical tools to review changes

  • look for poor variable names, spelling errors in comments, accidental addition or deletion of blank lines, complex code without comments, missing/extra spaces

  • fix everything you find before requesting feedback from others

  • run tests again “./m tests”

  • check for trailing spaces, line endings, template indentation “./m coding-std”

  • if Javascript files were changed, run https://www.jslint.com/

publish your change

  • do some final testing - practically and using the unit tests

  • commit your changes to your local repo, use a concise commit comment describing the change

    • while a commit message may have multiple lines, many tools show only 80 characters of the first line

    • stuff as much info as possible into those first 80 characters:

      <concise description of your change>, fixes #123

  • push the changeset to your public github repo

  • create a pull request so your changes will get reviewed and pulled into the main repository

  • if you fixed an issue from the issue tracker, be sure the issue gets closed after your fix has been pulled into main repo.

  • celebrate, loop back to “find a task to work on”

update your virtualenv

Every week or so, do “m quickinstall” to install new releases of dependent packages. If any new packages are installed, do a quick check for breakages by running tests, starting the build-in server, modify an item, etc.

MoinMoin architecture

moin2 is a WSGI application and uses:

  • flask as framework

    • flask cli and click for command line interface

    • flask-babel / babel / pytz for i18n/l10n

    • flask-theme for theme switching

    • flask-caching as cache storage abstraction

  • werkzeug for low level web/http page serving, debugging, builtin server, etc.

  • jinja2 for templating, such as the theme and user interface

  • flatland for form data processing

  • EmeraldTree for xml and tree processing

  • blinker for signalling

  • pygments for syntax highlighting

  • for stores: filesystem, sqlite3, sqlalchemy, memory

  • jquery javascript lib, a simple jQuery i18n plugin Plugin

  • CKeditor, the GUI editor for (x)html

  • TWikiDraw, AnyWikiDraw, svgdraw drawing tools

How MoinMoin works

This is a very high level overview about how moin works. If you would like to acquire a more in-depth understanding, please read the other docs and code.

WSGI application creation

First, the moin Flask application is created; see moin.app.create_app:

  • load the configuration (app.cfg)

  • register some modules that handle different parts of the functionality

    • moin.apps.frontend - most of what a normal user uses

    • moin.apps.admin - for admins

    • moin.apps.feed - feeds, e.g. atom

    • moin.apps.serve - serving some configurable static third party code

  • register before/after request handlers

  • initialize the cache (app.cache)

  • initialize index and storage (app.storage)

  • initialize the translation system

  • initialize theme support

This app is then given to a WSGI compatible server somehow and will be called by the server for each request for it.

Request processing

Let’s look at how it shows a wiki item:

  • the Flask app receives a GET request for /WikiItem

  • Flask’s routing rules determine that this request should be served by moin.apps.frontend.show_item.

  • Flask calls the before request handler of this module, which:

    • sets up the user as flaskg.user - an anonymous user or logged in user

    • initializes dicts/groups as flaskg.dicts, flaskg.groups

    • initializes jinja2 environment - templating

  • Flask then calls the handler function moin.apps.frontend.show_item, which:

    • creates an in-memory Item

      • by fetching the item of name “WikiItem” from storage

      • it looks at the contenttype of this item, which is stored in the metadata

      • it creates an appropriately typed Item instance, depending on the contenttype

    • calls Item._render_data() to determine what the rendered item looks like as HTML

    • renders the show_item.html template and returns the rendered item html

    • returns the result to Flask

  • Flask calls the after request handler which does some cleanup

  • Flask returns an appropriate response to the server

Storage

Moin supports different stores, like storing directly into files / directories, using key/value stores, using an SQL database etc, see moin.storage.stores. A store is extremely simple: store a value for a key and retrieve the value using the key + iteration over keys.

A backend is one layer above. It deals with objects that have metadata and data, see moin.storage.backends.

Above that, there is miscellaneous functionality in moin.storage.middleware for:

  • routing by namespace to some specific backend

  • indexing metadata and data + comfortable and fast index-based access, selection and search

  • protecting items by ACLs (Access Control Lists)

DOM based transformations

How does moin know what the HTML rendering of an item looks like?

Each Item has some contenttype that is stored in the metadata, also called the input contenttype. We also know what we want as output, also called the output contenttype.

Moin uses converters to transform the input data into the output data in multiple steps. It also has a registry that knows all converters and their supported input and output mimetypes / contenttypes.

For example, if the contenttype is text/x-moin-wiki;charset=utf-8, it will find that the input converter handling this is the one defined in converters.moinwiki_in. It then feeds the data of this item into this converter. The converter parses this input and creates an in-memory dom tree representation from it.

This dom tree is then transformed through multiple dom-to-dom converters for example:

  • link processing

  • include processing

  • smileys

  • macros

Finally, the dom-tree will reach the output converter, which will transform it into the desired output format, such as text/html.

This is just one example of a supported transformation. There are quite a few converters in moin.converters supporting different input formats, dom-dom transformations and output formats.

Templates and Themes

Moin uses jinja2 as its templating engine and Flask-Themes as a flask extension to support multiple themes. There is a moin/templates directory that contains a base set of templates designed for the Modernized theme. Other themes may override or add to the base templates with a directory named themes/<theme_name>/templates.

When rendering a template, the template is expanded within an environment of values it can use. In addition to this general environment, parameters can also be given directly to the render call.

Each theme has a static/css directory. Stylesheets for the Basic theme in MoinMoin are compiled using the source theme.less file in the Basic theme’s static/custom-less directory.

./m css  # Windows: m css

Internationalization in MoinMoin’s JS

Any string which has to be translated and used in the JavaScript code, has to be defined at moin/templates/dictionary.js. This dictionary is loaded when the page loads and the translation for any string can be received by passing it as a parameter to the _ function, also defined in the same file.

For example, if we add the following to i18n_dict in dictionary.js

"Delete this"  : "{{  _("Delete this") }}",

The translated version of “somestring” can be accessed in the JavaScript code by

var a = _("Delete this");

Testing

We use pytest for automated testing. It is currently automatically installed into your virtualenv as a dependency.

Running the tests

To run all the tests, the easiest way is to do:

./m tests  # windows:  m tests

To run selected tests, activate your virtual env and invoke pytest from the toplevel directory:

pytest --pep8  # run all tests, including pep8 checks
pytest -rs  # run all tests and output information about skipped tests
pytest -k somekeyword  # run the tests matching somekeyword only
pytest --pep8 -k pep8  # runs pep8 checks only
pytest sometests.py  # run the tests contained in sometests.py

Tests output

Most output is quite self-explanatory. The characters mean:

. test ran OK
s test was skipped
E error happened while running the test
F test failed
x test was expected to fail (xfail)

If something goes wrong, you will also see tracebacks in stdout/stderr.

Writing tests

Writing tests with pytest is easy and has little overhead. Just use the assert statements.

For more information, please read: https://docs.pytest.org/

IDE Setup

Most MoinMoin developers use PyCharm, either the Professional Edition or the Free Community Edition. Choose one or the other and follow the PyCharm setup instructions.

The screenshots below are from Windows 10, using Python 3.10 and PyCharm Community Edition to debug Moin2 code. *nix setup is similar.

Debug a Transaction

When setting up the Run/Debug Configurations, it is important to get the right values for the Script path, Parameters, Python interpreter, and Working directory. For general debugging of the moin2 code base those parameters should be similar to:

pycharm example

If the parameters are correct, then the Run dropdown menu will show green icons for run and debug. If the only choice under the Run menu is Edit Configuration, then one of the parameters is wrong, try again. Note: Py``Charm has a tendency to change the Working Directory field when other values are edited. Be sure it points to the repo root.

Once the configuration is correct, load a source program, set a break point and run the debugger. Point your browser to http://127.0.0.1:5000.

Debug a Moin Script

To debug one of the moin commands that are normally executed in a terminal window, follow the example below. You can view the list of moin commands by activating the virtual environment and doing a “moin –help”.

pycharm example

Debug a Test

To debug a test, start by going to the Py``Charm edit configuration view. Click the + in the upper left corner to show the popup list of configuration types. Choose Tox, and then follow the example below for other field values. Note the test starup will be rather slow, be patient.

pycharm example

Documentation

Sphinx (https://www.sphinx-doc.org) and reST markup are used for documenting moin. Documentation reST source code, example files and some other text files are located in the docs/ directory in the source tree.

Creating docs

Sphinx can create all kinds of documentation formats. The most common are the local HTML docs that are linked to under the User tab. To generate local docs:

./m docs  # Windows: m docs

Moin Shell

While the make.py utility provides a menu of the most frequently used commands, there may be an occasional need to access the moin shell directly:

source <path-to-venv>/bin/activate  # or ". activate"  windows: "activate"
moin -h                             # show help

Package Release on test.pypi.org

This procedure for updating test.pypi avoids adding release tags to master branch, hoping that someday there will be a real 2.0.0a1. Current state of moin 2 is pre-alpha.

Commit or stash all versioned changes. Pull all updates from master repo. Create a release branch. Run ./m quickinstall to update the venv and translations. Run tests. Add a tag with the next release number to the release branch:

git tag 2.0.0a14

Install or upgrade release tools:

pip install --upgrade setuptools wheel
pip install --upgrade twine
pip install --upgrade build

Build the distribution and upload to test.pypi.org:

py -m build > build.log 2>&1  # check build.log for errors
py -m twine upload --repository testpypi dist/*

Enter ID and password as requested.

Test Build

Create a new venv, install moin, create instance, start server, create item, modify and save an item:

<python> -m venv </path/to/new/virtual/environment>
cd </path/to/new/virtual/environment>
source bin/activate  # scripts\activate
pip install --upgrade pip  # next command fails with pip 9.0.1 and maybe later versions
pip install --pre --index-url https://test.pypi.org/simple --extra-index-url https://pypi.org/simple moin
moin create-instance --path <path/to/new/wikiconfig/dir>  # path optional, defaults to CWD
cd <path/to/new/wikiconfig/dir>  # skip if using default CWD
moin index-create
moin --help  # prove it works
moin run  # empty wiki
moin load-sample  # data but no index
moin index-build   # data with index
moin load-help -n en # load English help
moin load-help -n common # load help images

Announce update on #moin, moin-devel@python.org.