[CMS] Replace the markdown package with a CommonMark compliant one

[saroyanm]

Background

Our CMS currently uses the ​markdown package for rendering Markdown pages. While it works, we have encountered some difficulties in how certain things are rendered. As the format was never specified too well, different implementations often handle edge cases in completely different ways and the markdown package is no exception.

More recently ​the CommonMark initiative has come along, they are working at specifying the Markdown format clearly, so that all implementations can render the format consistently.

What to change

Investigate our options for a different, CommonMark compatible Markdown rendering module. Test for any differences in how any of our websites that use the CMS are rendered.

Notes

• CommonMark-py looks promising, but it is apparently ​about to change maintainer. Perhaps we should wait for that to happen before continuing?

Our problems with the current markdown package

To be continued...

[sebastian]

The rationale for this change, as outlined in the issue description at the moment, seem a little weak to me. The only limitation so far with markdown outlined in the issue description, isn't really a limitation but more like a configuration/usage issue. Note that markdown comes with a built-in extension that you can enable to ​allow markdown inside HTML elements.

Also upstream CommonMark only provides JS and C libraries. However, there is an ​inofficial and unmaintained Python port and a more recent ​fork of it. But the latter isn't even on PyPI yet (that means it needs to be installed manually from source). In fact they didn't even released a version yet, and don't seem to be a mature project. While on the other hand, the markdown module we currently use has been defacto-standard in the Python universe for years.

Also note that replacing the markdown rendering module, would require to re-test all pages showing content rendered from markdown.

[trev]

Ok, I was unaware of this extension. If it works for us then enabling it sounds like a simple solution for the immediate issue.

Generally, I would prefer CommonMark given that a clean specification and a test suite exist for it, and that specification is actually good enough for us without any non-standard extensions. However, with the CommonMark module changing maintainers right now we might be better off waiting with a decision until the new maintainers actually upload their version to PyPI. So this is a more long term project.

Supposedly, performance of the CommonMark module is better. This isn't something we need to believe however, we can test ourselves whether this improves performance in our case.

[kzar]

Once CommonMark-py's switch of maintainer has completed and the project has matured we can come back to this again. We all agree that it will be nice to use a CommonMark compliant Markdown library in the future.

For now we're going to just enable the markdown.extensions.extra extension, see #3354.

[kvas]

​CommonMark-py development has moved to ​Read the Docs organization, which seems like an appropriate and sustainable place to be. I think the maintainer issue can be considered resolved.

There's also another Python implementation called ​paka.cmark that uses CFFI around the reference implementation in C. We can see which one we prefer.

The spec is not final yet, but the authors are planning to release version 1.0 real soon now​this year.

Given all this, it might make sense to think about reviving this ticket, if it's still relevant. What do you think, @saroyanm, @juliandoucette?

[kvas]

Dear stakeholders of this ticket,

I'm cleaning up Sitescripts tickets in Trac due to its phase out. This is one of the tickets that I could not myself close or move, so I need your input on it.

Please let me know if this ticket is still relevant for you and we can then discuss where it should be moved. If you think that this ticket is no longer relevant, you can write a comment explaining this or just ignore this message.

If I see now comments in the ticket on October 14, I will close it as rejected.

Best regards,
Vasily